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PREFACE 


This  document  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  under  the 
task  order,  Ballistic  Missile  Defense  (BMD)  Software  Assessments,  and  fulfills  an  objective 
of  the  task,  to  prepare  “a  draft  report  providing  updated  lessons  and  guidance  for  conduct  of 
contractor  Software  Capability  Evaluations  in  the  BMD  program.” 

This  document  is  an  updated  version  of  an  earlier  IDA  study,  IDA  Paper  P-2771, 
Conducting  Software  Capability  Evaluations.  Since  then,  the  model  and  evaluation  process 
used  for  conducting  Software  Capability  Evaluations  have  evolved.  The  BMD  evaluation 
teams  have  performed  evaluations  for  the  Battle  Management  Command,  Control  and  Com¬ 
munications/Systems  Engineering  and  Integration  Program.  Consequently,  this  document 
was  written  to  provide  new  lessons  learned,  findings,  and  recommended  procedures  and 
techniques,  all  based  on  the  BMD  teams’  experiences  in  1995. 

This  document  was  reviewed  by  IDA  research  staff  members  Dr.  Richard  Ivanetich, 
Dr.  Judy  Popelas,  and  Mr.  David  A.  Wheeler.  Their  contributions,  and  in  particular  the  con¬ 
tributions  of  the  BMD  evaluation  teams,  are  gratefully  acknowledged. 
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Software  has  become  a  critical  element  in  Department  of  Defense  (DoD)  programs, 
and  the  DoD  has  increased  its  emphasis  on  the  need  to  evaluate  and  monitor  software  con¬ 
tractors  and  subcontractors.  Consequently,  the  Software  Engineering  Institute  (SEI)  was 
tasked  by  the  Air  Force  to  develop  a  software  evaluation  methodology;  the  results  included 
the  SEI  Capability  Maturity  Model  (CMM),  containing  five  levels  of  process  maturity  and 
their  associated  Key  Process  Areas  (KPAs),  and  the  evaluation  methodology  called  the 
Software  Capability  Evaluation  (SCE).  The  Ballistic  Missile  Defense  Organization 
(BMDO)  is  using  the  SCE  method  to  determine  whether  developers  have  good  software 
practices  that  help  avoid  cost  and  schedule  overruns.  In  conducting  SCEs,  the  BMDO  is 
encouraging  continuous  process  improvement  by  software  development  contractors.  SCEs 
have  been  scheduled  throughout  the  BMD  program  life  cycle,  and  program  offices  use 
SCEs  for  input  to  the  source  selection  process  as  well  as  for  monitoring  existing  contracts. 

Research  staff  members  from  the  Institute  for  Defense  Analyses  (IDA)  were  trained 
at  SEI  in  conducting  SCEs  and  participated  as  technical  advisors  for  the  SCEs  performed 
for  the  National  Test  Facility,  Brilliant  Pebbles,  and  Brilliant  Eyes  programs.  The  lessons 
learned  were  incorporated  in  an  earlier  study,  IDA  Paper  P-2771,  Conducting  Software 
Capability  Evaluations.  Since  then,  the  SEI  model  and  evaluation  process  have  evolved 
and  the  BMD  evaluation  teams  have  gained  additional  experience  by  conducting  SCEs  for 
the  BMDO  Battle  Management  Command,  Control,  and  Communications/Systems  Engi¬ 
neering  and  Integration  (BMC3/SE&I)  program.  This  document  is  an  update  of  P-2771, 
based  on  the  BMD  teams’  experience  in  1995.  It  describes  the  primary  team  activities  and 
responsibilities  in  conducting  SCEs  for  BMD  programs.  It  includes  specific  procedural 
guidance  for  each  activity,  and  key  artifacts  such  as  text  in  the  Request  for  Proposal  (RFP) 
for  informing  contractors  of  SCE  requirements.  Because  of  the  extensive  experience 
behind  these  recommendations,  other  DoD  programs  may  find  this  guide  helpful  for  their 
SCE  teams. 

The  content  of  this  guide,  in  terms  of  recommended  activities  and  guidance,  is  sum¬ 
marized  below  for  the  three  time  frames  of  the  evaluation  process,  i.e.,  before,  during,  and 
after  performing  the  SCE  team's  site  visits  to  the  contractors. 
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Activities  Prior  To  Conducting  Evaluations 

Develop  inputs  to  the  RFP.  Prospective  offerors  must  be  made  aware  of  the  SCE 
requirements  in  the  RFP.  Text  is  given  within  this  report  that  can  be  used  to  insert  the  SCE 
requirements  notice  within  the  RFP  and  the  associated  Instructions  for  Preparation  of  Pro¬ 
posals. 

Establish  evaluation  criteria.  The  Source  Selection  Authority  (SSA)  may  choose 
to  use  the  SCE  results  for  source  selection  either  as  a  specific  criterion,  general  consider¬ 
ation,  or  as  a  performance  risk  factor.  The  SSA  may  request  that  the  evaluation  results  sup¬ 
port  a  color  rating,  numerical  rating,  or  risk  rating.  Examples  of  these  rating  systems  are 
included  in  this  report. 

Identify  specific  program  needs.  Each  BMD  program  office  may  have  specialized 
software  evaluation  needs  not  currently  emphasized  in  the  CMM.  Additional  KPAs  may  be 
added  to  the  standard  SEI  CMM  or  the  evaluation  requirements  may  be  tailored  to  empha¬ 
size  program  office  needs.  A  list  of  questions  is  provided  to  help  tailor  the  CMM  to  meet 
these  special  needs. 

Notify  contractor  prior  to  evaluation.  When  performing  SCEs,  it  is  important  to 
notify  contractors  in  advance  how  the  SCEs  will  be  conducted.  During  the  pre-proposal 
conference  or  a  similar  coordination  meeting,  it  is  necessary  to  describe  the  evaluation  pro¬ 
cess,  pertinent  requirements  in  the  RFP,  site  visit  coordination  activities,  and  how  the 
results  will  be  used. 

Select  the  evaluation  team.  Detailed  qualifications  of  team  members  must  be  iden¬ 
tified.  Qualifications  include  training  and  technical  or  managerial  experience  in  the  area  of 
software  development  or  acquisition  support.  BMD  teams  should  not  consist  only  of  mem¬ 
bers  from  a  single  Service  or  Government  organization.  Assigning  team  members  from 
multiple  Service  organizations.  National  Laboratories,  and  Federally  Funded  Research  and 
Development  Centers  should  be  considered.  Other  considerations  include  team  skills,  lead¬ 
ership,  and  lack  of  conflict  of  interest.  All  SCE  team  members  will  be  required  to  sign  a 
Procurement  Integrity  Certificate  (PIC)  for  the  source  selection  at  hand.  The  PIC  requires 
disclosure  of  financial  interest  in  any  of  the  RFP  offerors.  Finally,  additional  training  should 
be  arranged  for  SCE  teams  who  have  little  SCE  experience  or  who  have  not  worked  togeth¬ 
er  on  an  actual  evaluation. 

Assign  KPAs  to  SCE  team  members.  Even  though  the  entire  SCE  team  is  respon¬ 
sible  for  understanding  and  evaluating  all  KPAs  under  review,  individual  team  members 
should  be  assigned  primary  responsibility  for  a  subset  of  KPAs.  For  each  assigned  KPA, 
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individuals  are  responsible  for  developing  interview  questions,  submitting  documentation 
requests,  interviewing  candidates,  and  generating  findings.  The  assignments  help  to  guar¬ 
antee  all  KPAs  are  adequately  and  consistendy  evaluated.  Included  is  a  list  of  KPA  assign¬ 
ments  used  by  a  previous  SCE  team. 

Estimate  SCE  expenses.  Due  to  the  extensive  travel  associated  with  conducting 
SCEs,  expenses  must  be  estimated  and  BMD  program  management  must  make  a  financial 
commitment  to  the  process.  Included  in  this  section  of  the  paper  is  a  list  of  actual  travel 
expenses  incurred  by  a  previous  SCE  team. 

Conducting  Evaluations 

Select  projects  to  be  evaluated.  The  contractor  will  provide  information  on  four  to 
six  projects.  The  SCE  team  will  select  three  projects  to  review  during  a  three-day  site  visit. 
The  projects  selected  must  adequately  represent  the  contractor’s  proposed  role  and  help  to 
judge  the  risk  associated  with  awarding  the  proposed  project  to  the  contractor.  Guidance 
for  selecting  the  appropriate  projects  is  provided. 

Establish  site  visit  schedule.  A  detailed  schedule  identifying  the  activities  during 
the  three-day  site  visit  is  described.  It  includes  a  list  of  interview  candidates,  a  prioritized 
list  of  topics  to  explore  during  each  interview,  and  the  allotted  time  for  document  reviews 
and  for  finalizing  SCE  findings  and  results. 

Submit  documentation  requests  to  the  contractor.  The  SCE  team  will  submit  a 
list  of  documents  to  be  made  available  to  the  SCE  team  prior  to  the  site  visit  and  during  the 
site  visit.  Prior  to  the  site  visit,  the  contractor  will  furnish  the  software  development  plan, 
project  profiles,  an  organization  chart,  and  a  completed  SEI  questionnaire  form.  This  doc¬ 
umentation  is  used  by  the  SCE  team  to  plan  an  interview  schedule,  identify  issues  and  ques¬ 
tions,  and  form  a  basis  for  the  findings.  During  the  site  visit,  additional  documentation  is 
requested  to  substantiate  findings.  A  detailed  list  of  documents  is  identified  for  each  KPA. 

Assemble  packing  list.  The  SCE  team  should  bring  a  set  of  reference  material, 
team  notes,  and  all  necessary  forms  to  be  used  while  on  site  at  the  contractor’s  facility. 

Develop  entrance  briefing.  The  SCE  team  and  the  contractor  will  each  provide  an 
entrance  briefing  to  identify  the  scope  of  the  SCE  and  the  contractor’s  software  process. 
Recommended  topics  for  the  presentations  are  listed  along  with  a  sample  set  of  slides. 

Notify  contractors.  Contractors  should  receive  a  detailed  notice  for  the  site  visit 
one  week  in  advance  of  the  SCE.  A  sample  notification  letter  used  by  previous  SCE  teams 
is  provided. 
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Establish  interview  approach.  Interviews  are  conducted  with  project  personnel, 
and  project  documentation  is  reviewed  to  verify  the  adequacy  of  the  contractor’s  software 
practices.  All  information  gathered  during  the  site  visit  will  be  documented  to  support  SCE 
findings.  Recommendations  are  included  for  conducting  interviews,  establishing  SCE  team 
member  roles,  documenting  interview  notes,  and  completing  specific  objectives  while  on 
site. 

Activities  After  Conducting  an  Evaluation 

Final  report.  The  SCE  team  will  prepare  a  report  of  its  findings  for  the  contractor, 
the  Source  Selection  Evaluation  Board  (SSEB),  the  BMD  program  office,  and  the  BMDO. 
The  report  will  be  labeled  Proprietary  Information  and  distribution  will  be  controlled. 
Descriptions  of  the  report  format  and  content  are  included  in  this  section. 

Contractor  feedback  of  evaluation  results.  The  winning  contractors  will  be 
briefed  after  the  contract  is  awarded;  at  this  time  the  contractors  can  provide  feedback. 
Guidance  is  included  for  providing  SCE  results  to  the  contractor. 

Use  of  evaluation  results  in  source  selection.  This  section  provides  guidance  for 
reporting  SCE  results  to  the  SSEB. 

Use  of  evaluation  results  for  contract  monitoring.  An  SCE  may  be  used  to  help 
monitor  a  contract.  The  program  office  can  compare  the  results  of  the  evaluation  with  the 
contractor’s  process  improvement  plans.  If  there  are  discrepancies,  the  program  office 
should  notify  the  contractor  to  produce  an  acceptable  plan  to  mitigate  the  risks  and  to 
reduce  the  principal  weaknesses  over  the  length  of  the  contract.  Other  approaches  for  mit¬ 
igating  risks  are  provided. 

Registry  of  evaluation  results.  The  results  of  an  SCE  should  be  stored  by  BMDO 
in  a  repository  for  possible  reuse  in  later  procurements.  This  can  reduce  time,  effort,  and 
expense.  Using  an  SCE  repository  could  also  shorten  the  procurement  cycle,  provided  pre¬ 
vious  results  are  used  and  the  number  of  new  SCEs  that  must  be  performed  is  reduced. 

Issues  Related  to  SCE  Findings.  Other  issues  associated  with  SCE  findings  are 
clarified  for  future  evaluations:  consistency  of  results,  evaluating  new  company  divisions, 
summarizing  KPA  findings,  weighting  SCE  results  in  a  teaming  arrangement,  establishing 
acceptable  levels  of  process  maturity,  and  saving  evidence. 
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1.1  Purpose 

This  paper  provides  the  means  for  improving  the  effectiveness  of  the  Ballistic  Mis¬ 
sile  Defense  (BMD)  Software  Capability  Evaluation  (SCE)  teams  and  the  quality  of  then- 
results.  The  originator  of  the  SCE  concept  and  process,  the  Software  Engineering  Institute 
(SEI),  provides  basic  information  for  performing  the  evaluations  in  the  SCE  Team  Mem¬ 
bers  Guide  and  the  associated  training  course  [SEI  1993a,  1994].  In  practice,  additional 
information  and  procedures  are  necessary  to  help  achieve  the  best  possible  results. 

The  SCE  is  a  valuable  tool  that  assists  in  ensuring  that  the  Federal  Government  gets 
a  timely  quality  product  for  its  software  investment.  SCEs  are  being  used  in  the  BMD  pro¬ 
gram  as  part  of  two  distinct  activities:  source  selection  and  contract  monitoring.  When  used 
in  source  selection,  SCE  results  figure  into  the  overall  scores  of  the  offerors.  When  SCEs 
are  used  for  contract  monitoring,  the  program  office  can  use  the  results  as  a  risk  manage¬ 
ment  indicator,  validate  whether  the  contractor’s  software  development  process  has  been 
maintained,  and  determine  whether  improvement  has  occurred. 

This  paper  is  for  use  within  the  BMD  program  by  teams  already  trained  to  conduct 
SCEs  by  SEI-licensed  trainers.  It  is  intended  to  supplement  the  training  material  and  reports 
with  other  information  and  procedures  the  BMD  teams  have  learned  and  used  through  their 
experiences  performing  SCEs.  In  particular,  this  paper  captures  the  lessons  learned  by  1 995 
BMD  SCE  teams  and  revises  lessons  which  were  documented  in  an  earlier  Institute  for 
Defense  Analyses  (IDA)  report  [Springsteen  1994].  The  previous  report  was  also  based  on 
an  earlier  version  of  the  evaluation  model  and  methodology  which  have  since  been  updated 
and  incorporated  into  BMD’s  SCE  practices. 

1.2  Background 

In  the  last  decade,  the  visibility  and  importance  of  software  in  Department  of 
Defense  (DoD)  programs  have  increased  the  need  to  improve  the  Federal  Government’s 
ability  to  evaluate  and  monitor  software  contractors.  Responding  to  a  request  by  the  United 
States  Air  Force,  SEI  developed  a  software  evaluation  methodology. 
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The  SEI  methodology  provides  five  levels  of  process  maturity,  each  associated  with 
key  process  areas  (KPAs).  Refer  to  Table  1  for  details  [SEI  1993b]. 

Table  1.  SEI  Capability  Maturity  Model 


Level 

Characteristics 

KPAs 

5 

Optimizing 

Continuous  process  improve¬ 
ment 

Process  Change  Management 
Technology  Change  Management 

Defect  Prevention 

4 

Managed 

Product  quality  planning  and 
tracking  of  measured  soft¬ 
ware  processes 

Software  Quality  Management 
Quantitative  Process  Management 

3 

Defined 

Developmentprocess  defined 
and  institutionalized  to  pro¬ 
vide  product  quality  control 

Peer  Reviews 

Intergroup  Coordination 

Software  Product  Engineering 

Integrated  Software  Management 
Training  Program 

Organization  Process  Definition 
Organization  Process  Focus 

2 

Repeatable 

Management  oversight  and 
tracking  of  project;  stable 
planning  and  product  base¬ 
lines 

Software  Configuration  Management 
Software  Quality  Assurance 

Software  Subcontract  Management 
Software  Project  Tracking  and  Over¬ 
sight 

Software  Project  Planning 

Requirements  Management 

1 

Initial 

Ad  hoc 

(unpredictable  and  chaotic) 

“People” 

BMD  evaluation  teams  attend  a  five-day  training  course  to  leam  how  to  apply  the 
basic  methods  for  conducting  an  SCE.  The  training  course  reviews  the  main  process  matu¬ 
rity  concepts,  teamwork  skills,  interview  techniques,  scoring  project  questionnaires,  and 
the  exit  briefing  containing  the  SCE  findings.  These  methods  are  taught  at  a  high  level  and 
are  documented  in  a  training  manual  [SEI  1994]. 

IDA  began  to  identify  better  practices  in  1991  after  participating  as  technical  advi¬ 
sors  for  the  SCEs  performed  at  the  National  Test  Facility  (NTF)  and  on  the  two  Brilliant 
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Pebbles  (BP)  contractors.  We  applied  the  lessons  learned  from  the  NTF  and  BP  SCEs  in 
subsequent  SCEs  performed  on  offerors  for  Brilliant  Eyes  (BE)  and  Battle  Management 
Command,  Control,  and  Communications/Systems  Engineering  and  Integration  (BMC3/ 
SE&I)  programs.  The  additional  lessons  collected  and  observations  made  during  these 
SCEs  are  used  as  the  basis  of  the  information  presented  in  this  document. 

1.3  Approach 

The  following  steps  were  taken  in  preparation  for  the  analyses  in  this  paper. 

a.  Trained  IDA  personnel  at  SEI. 

SEI  has  developed  and  administers  a  five-day  training  course  for  conducting  an 
SCE.  The  course  introduces  the  software  evaluation  methodology  that  focuses 
on  KPAs  tied  to  the  maturity  model.  SETs  case  studies  and  mock  evaluations 
are  used  to  provide  some  initial  hands-on  experience  to  the  trainees.  Currently, 
six  IDA  research  staff  members  have  completed  training. 

b.  Developed  supplemental  training  materials. 

Based  on  previous  experiences  in  conducting  SCEs  for  BMD,  IDA  reexamined 
SEI’s  training  course  and  training  materials  for  completeness  and  applicability 
to  the  BMD  program.  IDA  developed  additional  training  materials  to  emphasize 
the  information-gathering  aspects  of  conducting  an  SCE  and  to  share  the  les¬ 
sons  learned  from  earlier  SCEs.  A  course  was  administered  at  IDA  for  the  BE 
and  BMC3/SE&I  evaluation  teams  previously  trained  by  SEI  licensed  vendors. 

c.  Participated  in  conducting  evaluations. 

IDA  participated  in  over  14  SCEs  performed  in  support  of  BMD  programs,  spe¬ 
cifically  NTF,  BP,  BE,  and  BMC3/SE&I.  The  IDA  members  of  an  evaluation 
team  acted  as  technical  advisors,  providing  additional  depth  to  the  government 
team. 

d.  Developed  recommended  procedures  and  techruques. 

The  supplemental  training  materials  developed  by  IDA  and  the  lessons  learned 
by  the  various  evaluation  teams  have  been  collected  and  are  the  basis  for  this 
document. 
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1.4  Organization 

Section  1  presents  a  brief  background  of  the  origins  of  the  evaluation  method  and 
the  approach  taken  in  writing  this  paper.  Section  2  describes  the  views  of  the  Ballistic  Mis¬ 
sile  Defense  Organization  (BMDO)  on  SCEs  and  how  these  views  apply  to  the  BMD  pro¬ 
gram.  In  Sections  3, 4,  and  5,  suggestions  and  lessons  learned  are  presented  in  the  context 
of  the  activities  that  surround  an  SCE.  The  appendices  provide  plans,  worksheets,  and  sam¬ 
ple  findings  the  evaluation  teams  can  use  in  obtaining  the  SCE  results.  A  list  of  acronyms 
and  a  list  of  references  are  provided  at  the  end. 


2.  BMD  OBJECTIVES  IN  USING  SCEs 


To  help  ensure  that  BMD  software  developers  have  good  software  practices  and  to 
reduce  the  risk  of  software  cost  and  schedule  overruns,  SCEs  are  conducted  for  the  BMD 
program  using  the  Capability  Maturity  Model  (CMM).  BMDO’s  Software  Policy  3405 
establishes  the  basis  of  the  plans  and  requirements  to  use  SCEs  across  the  program. 

Since  the  CMM  is  limited  in  scope,  the  model  and  the  evaluation  criteria  may  be 
extended  to  help  evaluate  and  monitor  other  areas  of  importance  to  the  BMD  software  pro¬ 
gram.  Such  areas  include  trusted  software,  evolutionary  prototyping,  and  reusable  soft¬ 
ware.  This  section  of  the  report  describes  how  the  CMM  can  be  applied  within  the  BMD 
program  and  tailored  to  best  satisfy  the  program’s  needs. 

2.1  Use  the  Current  Evaluation  Method  and  Maturity  Model 

The  BMD  program  uses  the  current  SEI  evaluation  methodology  and  the  CMM 
[SEI  1993a,  1993c].  The  SCE  methodology  has  undergone  two  major  changes  in  the  last 
five  years  and  is  still  evolving.  As  SEI  methods  are  updated,  BMDO  plans  to  retrained  the 
BMD  teams  to  use  the  most  recent  version. 

The  first  SEI  process  maturity  model  was  documented  in  1987  by  Humphrey  and 
Sweet  [1987].  This  method  was  based  on  a  questionnaire  consisting  of  101  questions.  Since 
1991,  the  model  evolved  from  a  questionnaire  into  what  is  known  as  the  CMM  which 
includes  goals  and  practices  associated  with  KPAs  [SEI  1993c].  BMD  teams  plan  to  use  the 
most  current  version  of  the  maturity  model  as  it  continues  to  evolve. 

2.2  Encourage  Continuous  Process  Improvement 

The  underlying  goal  when  conducting  SCEs  is  to  encourage  continuous  process 
improvement  among  software  development  contractors.  It  should  be  an  on-going  effort 
within  a  contractor’s  organization  rather  than  something  done  only  once  to  satisfy  a  Source 
Selection  Evaluation  Board  (SSEB).  Continuous  process  improvement  will  not  simply  hap¬ 
pen  because  a  contractor  or  program  office  wishes  or  requires  it.  Rather,  a  comprehensive 
process  improvement  plan  should  be  put  in  place  and  followed. 
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Due  to  the  large  number  of  contractors  involved  in  the  BMD  program,  it  is  desirable 
to  use  both  the  SCE  and  the  software  process  assessments  (SPA)  to  encourage  continuous 
process  improvement.  SCEs  are  evaluations  performed  by  a  government  team  on  the  con¬ 
tractor’s  process,  whereas  a  SPA  is  performed  by  a  contractor  team  on  its  own  software  pro¬ 
cess.  Another  name  for  the  SPA  is  a  CMM  Based  Appraisal  -  Internal  Process  Improvement 
(CBA-IPI).  BMDO  Software  Directive  3405  specifies  an  evaluation  and  assessment  hier- 

archy,  as  pictured  in  Figure  L 


THAAD  Theatre  High  Altitude  Area  Defense  GBI  Ground  Based  Interceptor 

BMC3  Battle  Management  Command,  Control,  and  Communications  GBR  Ground  Based  Radar 

BMDO  Ballistic  Missile  Defense  Organization  PO  Program  Office 

FFRDC  Federally  Funded  Research  and  Development  Center  _ _ _ 

Figure  1.  BMD  Contract  Monitoring  Process 

The  Directive  3405  specifies  that  the  subcontractors  and  the  prime  contractors  per¬ 
form  annual  self-assessments  of  their  development  processes  and  develop  annual  software 
process  improvement  plans.  The  contractors  are  responsible  for  identifying  and  improving 
their  process  over  the  life  of  the  BMD  program.  In  addition,  prime  contractors  are  respon¬ 
sible  for  the  quality  and  cost  of  their  subcontractors’  software.  Thus,  the  prime  contractors 
are  encouraged  to  perform  annual  SCEs  on  their  subcontractors. 
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The  BMD  evaluation  team  may  only  evaluate  prime  contractors  and  not  all  of  the 
subcontractors.  But  the  BMD  teams  will  look  closely  at  how  well  the  prime  contractors 
oversee  their  subcontractors  and  will  validate  the  results  of  the  self-assessments  and  the 
quality  of  the  contractor’s  software  process  improvement  plans.  The  results  of  the  SCEs  are 
provided  to  the  contractor  and  the  element  program  managers  for  input  to  their  risk  man¬ 
agement  process. 


2.3  Schedule  SCEs  Throughout  BMD  Life  Cycle 

BMD  program  offices  will  use  SCEs  for  input  to  the  Source  Selection  Evaluation 
Board  (SSEB)  and  to  help  monitor  contracts  previously  awarded.  SCEs  will  be  used  for 
source  selection  for  both  DemonstrationA'alidation  (DenWal)  and  Engineering  Manufac¬ 
turing  and  Development  (EMD).  Since  it  may  take  one  to  three  years  for  a  contractor  to 
advance  from  one  maturity  level  to  another,  BMD  plans  include  the  use  of  SCEs  as  a  con¬ 
tract  monitoring  mechanism  to  encourage  the  contractors  to  continuously  improve  their 
software  development  process,  as  noted  in  Figure  2. 


Start 

Start 

DenWal 

EMD 

I  1-2  yrs  1-2  yrs  1-2  yrs 

1-2  yrs 

1  ^ 

Source  Contract 

Contract 

Soi 

irce 

1 

Contract 

1 

Contract 

Selection  Monitoring 

Monitoring 

Selection 

Monitoring 

Monitoring 

SCE  SCE 

SCE 

SCE 

SCE 

SCE 

Figure  2.  Schedule  for  SCEs 


The  &st  SCE  should  be  performed  during  source  selection,  the  next  SCE  one  to  two 
years  later  to  give  the  contractor  an  opportunity  to  improve  and  to  monitor  their  progress. 
Contractors  typically  welcome  SCEs  as  a  contract  monitoring  mechanism  since  it  gives 
them  an  independent  view  of  their  process  and  an  opportunity  to  prepare  for  the  SCE  that 
will  be  used  at  the  next  source  selection.  If,  however,  an  SCE  was  not  performed  during 
source  selection,  an  SCE  should  be  done  approximately  six  to  nine  months  after  the  con¬ 
tract  is  awarded,  as  noted  in  Figure  3.  By  this  time,  the  contractor  should  have  the  software 
process  defined  and  documented  in  a  software  development  plan  (SDP).  It  is  important  to 
have  the  SDP  available  prior  to  an  SCE  so  that  the  evaluation  team  can  verify  that  the  pro¬ 
cess  being  described  in  the  interviews  is  the  same  process  described  in  the  SDP  and  applied 
to  the  BMD  element. 


7 


Start 
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1 6-9  mo  1-2  yrs 


Start 

EMD 


1-2  yrs 


1-2  yrs 


1-2  yrs 


Contract  Contract  Source 

Monitoring  Monitoring  Selection 
SCE  SCE  SCE 


Contract  Contract 

Monitoring  Monitoring 
SCE  SCE 


Figure  3.  Alternative  Schedule  for  SCEs 


2.4  Emphasize  BMD  Projects  in  SCEs 

When  performing  an  SCE,  the  evaluation  team  looks  at  several  projects  within  the 
contractor’s  organization  to  gain  an  understanding  of  the  contractor’s  software  develop¬ 
ment  process.  The  SCE  team  reviews  information  on  four  to  six  of  the  contractor’s  projects 
and  interviews  people  from  three  projects.  Section  4.1  describes  in  more  detail  how  the 
team  selects  the  projects  to  review. 

In  general,  the  team  looks  at  several  projects  in  order  to  determine  what  processes 
are  unique  at  the  project  level  and  what  processes  are  standard  across  the  organization  and 
applied  to  all  of  the  projects.  When  new  software  projects  are  initiated,  it  is  desirable  to 
have  a  well-defined  organizational  approach  to  software  process  development  from  which 
a  new  software  project  can  draw.  The  organization’s  standard  software  process  should  be 
derived  and  refined,  based  on  the  experience  and  “lessons  learned”  from  the  projects  within 
the  organization.  It  is  not  desirable  for  each  project  to  learn  the  lessons  first  hand,  but  rather 
to  learn  from  the  trial  and  error  of  previous  projects.  Thus,  an  SCE  evaluates  the  organiza¬ 
tion’s  approach  to  software  development  by  looking  at  the  organizational  practices  and  how 
they  are  used  in  on-going  projects. 

When  a  project  is  initiated,  the  organization’s  standard  approach  to  software  devel¬ 
opment  has  much  more  effect  than  when  a  project  is  nearing  completion.  Consequently,  the 
focus  of  a  BMD  SCE  will  vary,  depending  on  whether  the  BMD  project  is  just  being  initi¬ 
ated  or  whether  it  is  well  into  development.  For  example,  at  the  start  of  DemA^al,  the  off¬ 
erors  will  not  have  a  well-defined  software  process  for  the  BMD  element  for  which  they 
are  competing.  An  SCE  for  DenVVal  source  selection  should  therefore  evaluate  the  pro¬ 
cesses  being  applied  to  other  projects  within  the  contractor’s  organization  supporting 
BMD.  Once  the  DenVVal  contract  is  awarded,  the  contractor  will  begin  to  define  the  soft- 
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ware  process  for  the  BMD  element.  By  the  time  DenVVal  is  completed,  the  process  being 
applied  to  the  HMD  contract  is  well  defined  for  the  BMD  element.  The  SCE  performed  for 
EMD  source  selection  will  focus  less  on  the  organization’s  process  and  more  on  the  process 
being  used  on  the  BMD  element. 

2.5  Extend  the  Capability  Maturity  Model 

The  KPAs  in  SETs  CMM  cover  certain  components  that  are  recognized  as  good 
software  development  practice.  BMDO’s  efforts  to  improve  the  development  process  fur¬ 
ther  raise  additional  requirements  that  could  be  included  in  SCEs  such  as  the  BMD  Trusted 
Software  Methodology  (TSM)  and  use  of  a  common  information  architecture.  The  basic 
approach  for  extending  the  CMM  into  additional  areas  involves  tailoring  the  existing  CMM 
KPAs  or  adding  other  KPAs  to  address  the  unique  requirements.  For  reference,  these  may 
be  called  program-defined  KPAs,  in  contrast  to  SEI-defined  KPAs  included  in  the  CMM. 
Refer  to  Section  3.3  for  information  on  tailoring  the  existing  KPAs  to  satisfy  specific  pro¬ 
gram  needs. 

When  adding  program-defined  KPAs,  it  is  most  important  to  preserve  intact  SEl’s 
model  and  evaluation  approach  for  several  reasons.  First,  this  allows  a  program  to  leverage 
off  SEI-developed  training  experience  applying  SEI’s  approach.  Second,  a  program  can  use 
the  benefit  data  emerging  from  this  experience  as  an  aid  to  assessing  its  own  improvement 
achievements  or  problems.  Third,  a  program  should  find  it  expeditious  and  economical  to 
address  the  added  requirements  with  the  same  team  and  during  the  same  SCE  site  visit. 

The  addition  of  a  program-defined  KPA,  such  as  for  the  BMD  TSM,  involves  devel¬ 
oping  a  set  of  goals  and  practices  for  satisfying  the  KPA.  The  goals  may  be  flexible  and  per¬ 
mit  multiple  ways  of  satisfying  the  KPA.  They  may  be  expressed  in  terms  of  trust  principles 
and  supported  with  a  set  of  candidate  interview  questions  to  be  answered  through  the  SCE 
interviews.  The  criteria  also  must  lead  to  a  report  for  the  added  KPA,  similar  to  the  one  pre¬ 
pared  for  SEI’s  KPAs;  that  is,  it  must  help  identify  strengths  and  weaknesses,  and  support 
a  clear  resolution  of  whether  or  not  the  KPA  is  satisfied. 

Any  score  or  level  scale  associated  with  program-defined  KPAs  should  be  separate 
and  distinct  from  SEI  maturity  levels  to  preserve  SEI’s  approach  without  change.  Strengths, 
weaknesses,  and  KPA  satisfaction  should  be  the  important  consideration  in  either  case,  not 
overall  score  or  maturity  level. 

A  program-defined  KPA  may  involve  requirements  in  which  SCE  team  members 
are  not  well  versed.  This  means  that  special  training  may  have  to  be  established  for  SCE 


teams,  so  that  they  can  consistently  judge  various  implementations  of  requirements  they 
will  encounter  in  practice. 

Program-defined  KPAs  may  depend  in  part  upon  evidence  and  requirements  estab¬ 
lished  for  CMM  KPAs.  An  SCE  team  should  have  this  overlap  clearly  in  mind  during  inter¬ 
views  in  order  to  gather  all  the  pertinent  facts  at  the  most  convenient  time,  rather  than  doing 
repeat  interviews  to  handle  added  KPAs  separately  from  the  CMM. 

Other  evaluation  methods  such  as  Software  Development  Capability/Capacity 
Review  (SDC/CR)  [AFSC  1991]  and  Software  Productivity  Research  (SPR)  [SPR  1991] 
also  show  that  the  CMM  is  not  exhaustive  and  could  be  extended  further  to  explore  other 
potential  risk  areas.  For  example,  additional  areas  of  investigation  in  SDC/CR  not  found  in 
the  CMM  KPAs  are  systems  engineering  and  development  tools.  SPR  has  additional  areas 
of  coverage  such  as  the  physical  environment,  experience  of  the  staff,  and  development 
methodologies. 

An  SCE  team  must  be  able  to  address  all  added  KPAs  within  a  reasonable  additional 
time  on  site,  perhaps  no  more  than  one  additional  day.  This  factor  especially  limits  the  num¬ 
ber  and  scope  of  additional  requirements.  Experience  with  added  KPAs  is  still  needed  to 
provide  firm  guidelines  on  scheduling.  For  specific  guidance  on  extending  the  CMM  to 
include  the  TSM  requirements  refer  to  [Springsteen  1993]. 


3.  ACTIVITIES  PRIOR  TO  CONDUCTING  EVALUATIONS 


Prior  to  performing  SCEs,  several  activities  need  to  be  accomplished  so  that  the 
contractor  and  government  program  office  can  incorporate  the  SCEs  into  their  schedules 
and  budgets.  In  order  of  occurrence,  they  are  1)  develop  inputs  to  the  Request  for  Proposal 
(RFP),  2)  establish  evaluation  criteria,  3)  identify  specific  program  needs,  4)  notify  contrac¬ 
tor  prior  to  the  evaluation,  5)  select  the  evaluation  team,  6)  assign  KPAs  to  SCE  team  mem¬ 
bers,  and  7)  estimate  expenses.  This  section  will  discuss  these  activities  in  more  detail. 

3.1  Develop  Inputs  to  RFP 

When  using  SCEs  for  source  selection  and  contract  monitoring,  the  offerors  must 
be  made  aware  of  the  requirement  in  the  RFP.  Appendix  A  of  this  document  provides  text 
that  can  be  used  to  insert  the  SCE  requirements  notice  within  the  RFP,  specifically.  Sections 
L  and  M  of  the  RFP.  The  text  is  expected  to  be  tailored  to  accommodate  the  specific  require¬ 
ments  of  the  acquisition. 

Appendices  B  and  C  of  this  document  provide  additional  information  that  should 
be  included  in  the  RFP.  Appendix  B  is  a  sample  project  profile  that  each  contractor  should 
complete.  The  project  profile  requests  general  information  about  a  software  development 
effort,  such  as  coding  language  used,  host  development  system,  and  applicable  standards. 
Appendix  C  is  a  form  for  recording  the  answers  to  the  SEI  Maturity  Questionnaire.  The 
questionnaire  requests  detailed  information  on  the  software  engineering  practices  used  on 
a  project.  Each  of  the  contractors  should  use  this  form  to  record  the  answers  for  four  to  six 
projects.  The  SCE  team  will  then  review  both  the  project  profiles  and  the  questionnaire 
responses  to  select  the  set  of  projects  to  examine  during  the  site  visit. 

3.2  Establish  Evaluation  Criteria 

The  Source  Selection  Authority  (SSA)  will  use  the  results  of  an  SCE  during  source 
selection  as  either  a  specific  criterion,  general  consideration,  or  a  performance  risk.  A  spe¬ 
cific  criterion  is  preferred  since  the  SCE  process  can  provide  valuable  information  useful 
in  selecting  a  contractor.  In  any  case,  the  SSA  will  request  that  evaluation  criteria  be  estab- 
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lished  for  using  SCE  results  to  support  a  color,  numerical,  or  risk  rating,  depending  on 
whether  the  procurement  is  for  the  Air  Force  or  the  Army. 

These  rating  systems  can  be  applied  to  the  results  of  aU  the  KPAs  combined  or  sin¬ 
gularly.  When  the  KPA  results  are  combined,  the  rating  should  identify  the  offeror  as  high 
risk  if  it  has  a  low  process  maturity,  and  low  risk  if  the  maturity  rating  is  high.  In  other 
words,  contractors  with  a  level  1  maturity  would  be  ranked  lower  than  those  with  a  level  3 
maturity.  Instead  of  applying  the  rating  systems  to  a  maturity  level,  they  can  be  applied  to 
each  KPA  individually.  For  example,  an  offeror’s  ability  to  satisfy  all  or  some  of  the  goals 
of  a  KPA  can  be  used  to  assign  a  color,  number,  or  risk  rating  to  each  KPA. 

The  following  criteria  may  be  used  to  map  overall  SCE  results  to  the  Air  Force  s 
color  rating  scheme: 

.  Blue:  A  blue  rating  is  given  when  the  SCE  findings  show  that  the  offeror  is 
acceptable  in  all  level  2  and  level  3  KPAs. 

•  Green:  A  green  rating  is  given  when  the  SCE  findings  show  the  offeror  is 
acceptable  in  all  level  2  KPAs.  In  addition,  the  offeror  is  acceptable  in  at  least 
three  of  the  following  level  3  KPAs:  Organization  Process  Focus,  Organization 
Process  Definition,  Peer  Reviews,  Intergroup  Coordination,  Software  Product 
Engineering,  Training  Program. 

*  Yellow:  A  yellow  rating  is  given  when  the  SCE  findings  show  the  offeror  is 
acceptable  in  at  least  four  of  the  following  level  2  KPAs:  Software  Project  Plan¬ 
ning,  Software  Project  Tracking  and  Oversight,  Software  Configuration  Man¬ 
agement,  Software  Quality  Assurance,  Requirements  Management,  Software 
Subcontract  Management. 

•  Red:  The  red  rating  is  given  when  the  SCE  findings  show  the  offeror  is  accept¬ 
able  in  fewer  than  four  of  the  following  level  2  KPAs:  Software  Project  Plan¬ 
ning,  Software  Project  Tracking  and  Oversight,  Software  Configuration 
Management,  Software  Quality  Assurance,  Requirements  Management,  Soft¬ 
ware  Subcontract  Management. 

As  an  alternative,  the  following  numerical  rating  method  might  be  used.  In  this 
example,  a  total  of  13  points  can  be  earned  as  follows: 

*  For  each  of  the  following  level  2  KPAs  that  are  acceptable,  the  offeror  earns  a 
point:  Software  Project  Planning,  Software  Project  Tracking  and  Oversight, 
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Software  Configuration  Management,  Software  Quality  Assurance,  Require¬ 
ments  Management,  Software  Subcontract  Management. 

♦  For  the  offeror  to  earn  any  additional  points,  the  offeror  must  have  been  accept¬ 
able  in  at  least  four  of  the  level  3  KPAs  identified  above.  The  offeror  can  earn 
an  additional  point  for  the  following  KPAs,  provided  that  they  were  acceptable. 
Organization  Process  Focus,  Organization  Process  Definition,  Peer  Reviews, 
Intergroup  Coordination,  Integrated  Software  Management,  Software  Product 
Engineering,  and  Training  Program. 

The  SSEB  could  apply  one  of  these  rating  schemes  to  the  SCE  results.  The  exam¬ 
ples  given  previously  may  be  tailored  to  meet  the  specific  acquisition  needs.  If  the  program 
office  is  selecting  between  two  contractors  that  have  similar  maturity  levels,  the  criteria 
should  be  more  stringent  to  differentiate  between  them.  Depending  on  the  program  office’s 
concerns,  the  evaluation  criteria  can  be  tailored  to  emphasize  specific  KPAs. 

3.3  Identify  Specific  Program  Needs 

Each  BMD  element  program  office  may  have  specialized  software  needs.  It  is  con¬ 
ceivable  that  the  specialized  needs  of  one  or  more  of  the  elements  would  not  be  appropri¬ 
ately  addressed  in  an  SCE  performed  with  the  standard  CMM-based  KPA  set.  The  program 
office  will  need  to  determine  if  additional  KPAs  need  to  be  added  or  if  the  existing  require¬ 
ments  need  to  be  tailored. 

Following  is  a  list  of  sample  questions  to  ask  the  program  manager  when  tailoring 
the  scope  of  the  existing  KPAs.  In  parentheses  are  the  KPAs  that  would  be  affected  by  the 
response  to  the  question. 

a.  Are  specific  cost  or  schedule  reports  required  of  the  contractor  (e.g.,  specific 
cost,  schedule,  accounting  reports)?  (Software  Project  Tracking  and  Oversight 
KPA) 

b.  Is  the  RFP  requiring  the  offerors  to  use  advanced  software  engineering  tools  and 
techniques  (e.g.,  object-oriented  development,  commercial-off-the-shelf  inte¬ 
gration,  software  reuse)?  (Software  Product  Engineering,  Organization  Process 
Focus  KPAs) 

c.  Is  a  particular  software  life  cycle  model  being  mandated  (e.g.,  evolutionary  pro¬ 
totyping  vs.  full-scale  development)?  (Software  Project  Planning  and  Organiza¬ 
tion  Process  Definition  KPAs) 
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d.  What  level  of  involvement  will  the  customer  have  with  the  contractors  (e.g., 
periodic  formal  reviews  vs.  integrated  product  teams)?  (Requirements  Manage¬ 
ment  and  Intergroup  Coordination  KPAs) 

e.  Who  will  be  responsible  for  performing  Software  Configuration  Management 
and  Software  Quality  Assurance  (e.g.,  prime  contractor,  subcontractor,  or  gov¬ 
ernment)?  (Software  Configuration  Management  and  Software  Quality  Assur¬ 
ance  KPAs) 

f.  Will  subcontractors  be  used  to  develop  software?  (Software  Subcontract  Man¬ 
agement  KPA  can  be  eliminated  if  answer  is  negative.) 

Depending  on  the  level  of  importance,  these  requirements  can  be  added  as  separate 
KPAs  or  emphasized  within  the  scope  of  existing  KPAs.  If  given  the  opportunity  during  a 
source  selection  process,  SCE  teams  could  review  proposals  to  determine  what  methods, 
tools,  and  processes  were  being  proposed  and  target  them  during  the  on-site  evaluation. 
Usually,  however,  there  is  no  time  for  the  SCE  team  to  evaluate  proposals  since  the  evalu¬ 
ation  and  contract  award  schedule  is  too  demanding  [Ragan  1995]. 

In  addition  to  adding  requirements,  the  program  manager  should  be  interviewed  to 
determine  if  any  of  the  KPAs  can  be  deleted  from  the  scope  of  the  evaluation  process.  Due 
to  the  time  constraints,  it  is  not  recommended  that  the  SCE  team  evaluate  all  of  the  level  2 
and  level  3  KPAs  in  the  three-day  evaluation  period.  It  is  too  difficult  for  the  team  to  scru¬ 
tinize  all  13  KPAs  to  a  significant  level  of  detail.  Based  on  the  experience  of  the  previous 
BMD  SCE  teams,  there  are  certain  KPAs  which  all  of  the  contractors  have  passed  and  for 
which  significant  weaknesses  have  not  been  identified,  i.e..  Software  Configuration  Man¬ 
agement  and  Software  Quality  Assurance.  At  minimum,  it  is  recommended  that  these  KPAs 
be  eliminated  from  the  SCE  review  process  so  that  the  team  may  perform  a  more  thorough 
evaluation  on  the  remaining  areas  of  interest. 

3.4  Notify  Contractor  Prior  to  Evaluation 

Prior  to  an  SCE,  it  is  important  to  meet  with  a  contractor  to  formally  notify  them  of 
the  SCE  requirements  and  describe  the  general  evaluation  process  that  BMD  will  use. 

During  a  source  selection,  all  potential  offerors  will  be  notified  through  the  RFP  and 
the  pre-proposal  conference  that  SCEs  wHl  be  conducted.  Some  offerors  may  not  be  famil¬ 
iar  with  the  CMM  and  the  SCE  process;  hence  it  is  important  to  provide  a  brief  overview 
during  the  pre-proposal  conference.  The  overview  should  include  a  general  description  of 
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the  evaluation  process,  the  pertinent  requirements  in  the  RFP,  site  visit  coordination  activ¬ 
ities,  and  a  description  of  how  the  SCE  results  will  be  used  in  the  source  selection  process. 
Refer  to  Appendix  D  for  a  sample  of  the  slides  that  were  used  at  earlier  pre-proposal  con¬ 
ferences.  These  same  slides  apply  when  coordinating  an  SCE  for  the  purposes  of  monitor¬ 
ing  a  contract.  It  is  recommended  that  the  coordination  meeting  be  held  approximately 
three  months  in  advance  of  the  SCE. 

3.5  Select  the  Evaluation  Team 

When  forming  an  SCE  team,  the  program  office  should  coordinate  through  BMDO 
which  is  responsible  for  SCE  team  training  and  schedules.  There  are  several  qualification 
requirements  that  BMDO  requires  of  the  team  and  its  members. 

All  team  members  must  have  attended  an  SCE  training  course,  preferably  together. 
Currently,  SEI  has  certified  vendors  to  provide  a  five-day  training  course  for  first  time  eval¬ 
uators.  A  two-day  refresher  training  course  is  available  for  individuals  who  were  previously 
trained  on  earlier  versions  of  the  SCE  method. 

Before  being  selected  for  SCE  training,  potential  trainees  must  have  adequate  soft¬ 
ware  technical  or  managerial  experience.  SEI  recommends  that  trainees  have  at  least  seven 
years  of  software  development  or  acquisition  experience.  It  is  not  adequate  to  have  a  SCE 
team  consist  solely  of  software  acquisition  experts  or  development  experts.  It  is  desirable 
to  have  a  mix  of  both  professions  on  the  team;  at  least  one  representative  must  have  exten¬ 
sive  software  acquisition  experience  and  at  least  three  representatives  must  have  software 
development  experience.  When  deciding  the  team  composition,  it  is  also  important  that  at 
least  two  of  the  members  have  a  strong  background  in  each  of  SEI’s  KPAs. 

BMD  teams  should  not  consist  of  members  from  a  single  Service  or  government 
organization.  Representatives  from  the  Air  Force,  Army,  NTF,  and  Federally  Funded 
Research  and  Development  Centers  (FFRDCs)  should  be  considered  for  the  team. 

Other  considerations  for  selecting  SCE  team  members  include  team  skills,  leader¬ 
ship,  and  lack  of  conflict  of  interest.  Team  skills  are  an  important  aspect  during  an  evalua¬ 
tion.  Those  who  find  the  consensus  process  difficult  or  who  are  unable  to  contribute  to  the 
SCE  process  will  not  be  effective  SCE  team  members.  Team  members  must  have  good 
communication  skills  in  order  to  work  with  other  team  members  and  with  contractors  dur¬ 
ing  the  evaluation  process.  Team  members  must  be  good  listeners  so  that  they  can  judge 
what  they  hear  during  the  evaluation  process.  Team  members  should  also  take  initiative. 
Without  such  initiative,  the  evaluation  process  might  become  shallow  and  superfluous.  It  is 


15 


also  important  to  have  some  team  members  who  can  take  a  leadership  role  during  an  SCE 
to  ensure  that  the  SCE  progresses  smoothly  and  effectively. 

All  SCE  team  members  will  be  required  to  sign  a  Procurement  Integrity  Certificate 
(PIC)  for  the  source  selection  at  hand.  The  SCE  team  leader  with  the  assistance  of  the  pro¬ 
gram  office  should  check  to  ensure  that  all  potential  SCE  team  members  will  be  able  to  sign 
the  PIC.  (Among  other  things,  the  PIC  requires  disclosure  of  financial  interest  in  any  of  the 
RFP  offerors.)  PICs  should  be  signed  and  returned  to  the  contracting  officer  within  the  pro¬ 
gram  office  prior  to  the  start  of  the  SCE  activities. 

If  the  chosen  SCE  team  has  little  previous  SCE  experience  or  has  not  worked 
together  on  an  actual  evaluation,  it  is  beneficial  to  arrange  additional  training.  The  purpose 
of  such  training  is  to  sharpen  skills  learned  during  formal  SCE  training  and  to  develop  inter¬ 
personal  team  building  skills.  The  CMM  and  the  SCE  process  discussed  during  the  SCE 
training  course  should  be  reviewed.  In  particular,  each  KPA  should  be  reviewed  to  famil¬ 
iarize  team  members  with  the  criteria  to  be  used  during  the  evaluation.  If  possible,  a  prac¬ 
tice  SCE  should  be  arranged,  consisting  of  interviews,  documentation  reviews,  and  the 
team  consensus  process.  If  a  practice  SCE  is  unmanageable  given  the  time  and  resource 
commitments,  a  mock  SCE  should  be  arranged  with  “typical”  people  encountered  during 
site  visits. 

3.6  Assign  KPAs  to  SCE  Team  Members 

Due  to  the  scope  of  the  CMM  and  the  time  constraint  of  the  SCE  process,  KPAs 
should  be  divided  amongst  the  SCE  team  members.  Prior  to  the  start  of  an  SCE,  each  team 
member  at  a  minimum  is  responsible  for  thoroughly  understanding  his  or  her  assigned 
KPAs,  developing  interview  questions,  and  listing  potential  documents  to  evaluate  once  on 
site.  During  the  evaluation,  each  team  member  will  be  responsible  for  interviewing  candi¬ 
dates  to  better  understand  the  contractor’s  ability  to  address  each  KPA  goal,  reviewing  doc¬ 
uments  which  substantiate  that  the  process  is  being  used,  and  generating  findings  in  terms 
of  strengths  and  weaknesses.  While  a  team  member  may  have  been  assigned  a  primary  set 
of  KPAs,  all  team  members  should  be  prepared  to  support  the  other  team  members  in  their 
evaluation  of  all  the  KPAs. 

When  assigning  KPAs  to  team  members,  it  is  important  to  understand  which  KPAs 
individuals  feel  most  knowledgeable  about.  There  are  logical  grouping  of  KPAs  that  will 
help  establish  continuity  among  team  members  and  help  to  define  their  areas  of  responsi- 
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bility.  Following  is  a  list  of  the  KPA  groups  that  were  assigned  to  a  five-member  team  dur¬ 
ing  previous  BMD  SCEs; 

a.  Software  Project  Planning,  Software  Project  Tracking  and  Oversight,  Integrated 
Software  Management 

b.  Requirements  Management,  Software  Product  Engineering 

c.  Software  Quality  Assurance,  Software  Configuration  Management,  Training 
Program 

d.  Organization  Process  Focus,  Organization  Process  Definition 

e.  Software  Subcontract  Management,  Intergroup  Coordination,  Peer  Review 

3.7  Estimate  SCE  Expenses 

The  program  manager  and  BMDO  should  be  aware  of  the  anticipated  expenses  for 
conducting  SCEs.  There  are  expenses  associated  with  team  preparation,  site  visits,  and  final 
report  preparation.  Assuming  BMDO  already  funds  the  labor  associated  with  each  team 
member,  the  added  expenses  are  generally  travel  related.  Table  2  contains  a  list  of  the  actual 
expenses  incurred  by  an  SCE  evaluation  team  to  perform  one  contractor  evaluation. 


Table  2.  Example  of  SCE  Travel  Expenses 


Category 

Range  of 
Expenses  ($) 

Quantity 
per  Person 

Average 
Expense 
per  Person  ($) 

Total  Expense  per 
Team 

(5  Members)($) 

Airfare 

436-1,432 

1  round  trip 

949 

4,745 

Hotel 

42-102 

4  nights 

288 

1,440 

Per  diem 

26-38 

5  days 

160 

800 

Car  rental 

36-42 

4  days 

156 

780 

Miscellaneous: 

90 

450 

Hotel  parking 

5 

4  days 

Phone  calls 

10 

1 

Local  mileage 

15 

1 

Airport  parking 

30 

1 

Personal  gas 

15 

1 

TOTAL 

1,643 

8,215 
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4.  CONDUCTING  EVALUATIONS 


This  section  provides  guidance  to  an  SCE  team  for  the  period  between  when  RFP 
responses  are  received  and  the  team  has  completed  the  three-day  on-site  evaluation  of  each 
contractor.  In  order  of  sequence,  the  SCE  team  will  1)  select  projects  to  be  evaluated,  2) 
establish  site  visit  schedule,  3)  submit  documentation  requests  to  the  contractor,  4)  assem¬ 
ble  packing  list,  5)  develop  site  visit  entrance  briefings,  6)  notify  contractors,  and  7)  estab¬ 
lish  interview  approach.  This  section  provides  additional  details  on  each  of  these  activities. 

4.1  Select  Projects  To  Be  Evaluated 

In  response  to  the  RFP,  a  contractor  will  provide  information  on  four  to  six  projects. 
From  these  projects,  the  SCE  team  selects  three  projects  it  wishes  to  examine  during  the 
SCE.  The  SCE  team  will  use  information  contained  in  both  project  profiles  and  the  SEI 
questionnaire  to  select  the  projects  to  be  evaluated  in  greater  detail  during  the  site  visit.  This 
selection  process  is  critical  to  the  success  of  an  SCE.  The  projects  selected  must  provide 
data  that  can  be  used  to  judge  the  risk  associated  with  awarding  the  proposed  project  to  the 
contractor.  The  following  paragraphs  provide  guidance  for  selecting  the  projects. 

Projects  examined  during  an  SCE  must  be  from  the  same  site,  division,  group  or 
profit  center  as  that  of  the  proposed  project.  These  organizational  divisions  vary  consider¬ 
ably  in  industry.  During  an  evaluation,  the  team  will  be  looking  for  an  organization-level 
set  of  policies  and  procedures  that  are  applied  consistently  across  all  projects.  Specifically, 
the  team  will  seek  to  examine  projects  with  a  common  software  quality  asstnance  (SQA), 
software  configuration  management  (SCM),  and  software  engineering  process  group 
(SEPG).  One  way  to  determine  whether  the  selected  projects  are  appropriate  for  examina¬ 
tion  is  to  trace  the  management  control  from  each  of  these  functions  up  through  the  orga¬ 
nization.  These  lines  should  converge  at  a  point  also  in  the  management  structure  of  the 
proposed  project;  i.e.,  the  SQA  manager,  SCM  manager,  and  SEPG  managers  should  be  the 
same  for  all  the  projects  being  reviewed. 

Contractor  responsibilities  on  the  selected  projects  should  be  similar  as  those  on  the 
proposed  project.  If  the  contractor  is  the  prime  on  the  proposed  project,  it  is  important  that 
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the  projects  selected  for  examination  represent  examples  where  the  contractor  also  served 
as  the  prime.  This  will  allow  the  SCE  team  to  evaluate  the  contractor’s  ability  to  assess  and 
guide  the  processes  used  by  the  subcontractor.  If  the  contractor  has  never  been  the  prime 
before  but  will  be  in  the  proposed  contract,  this  is  a  risk  that  must  be  brought  to  the  attention 

of  the  SSA. 

In  addition,  projects  are  less  suitable  for  an  SCE  if  they  were  subcontracted  from 
another  prime  contractor,  used  substantial  government-furnished  software,  or  were  devel¬ 
oped  as  an  internal  research  and  development  (IRAD)  project.  Projects  subcontracted  from 
another  contractor  are  not  good  candidates  since  the  development  processes  used  by  the 
subcontractor  may  have  been  influenced  and  guided  by  the  prime  contractor  and  do  not  rep¬ 
resent  those  of  the  subcontractor.  Similarly,  the  SCE  team  should  avoid  evaluating  projects 
where  most  of  the  software  was  government  furnished  since  the  team  is  evaluating  the  con¬ 
tractor’s  development  process  rather  than  the  ability  of  the  contractor  to  integrate  govern¬ 
ment-furnished  software.  IRAD  software  projects  are  developed  internal  to  the  contractor 
with  no  government  interface.  The  project  management  and  development  practices  may  not 
be  representative  of  a  larger  government-funded  project. 

Projects  selected  for  examination  during  an  SCE  should  be  technically  related  to  the 
proposed  project.  For  example,  a  management  information  system  (MIS)  project  may  not 
provide  adequate  information  for  judging  the  risk  involved  with  building  a  sophisticated 
launch  control  system. 

The  scale  of  the  development  effort  on  the  selected  projects  should  be  roughly 
equivalent  to  that  of  the  proposed  project.  Staff  resources  and  lines  of  somce  code  are  two 
possible  indicators  of  the  scale  of  the  projects.  Despite  guidance  provided  to  the  contractor, 
the  team  may  find  wide-ranging  projects  offered  for  their  consideration. 

It  is  preferable  that  the  projects  selected  for  examination  be  on-going  development 
projects.  At  worst  the  projects  may  be  up  to  six  months  into  the  maintenance  phase.  After 
a  project  is  completed  or  is  in  the  advanced  maintenance  phases,  people,  documentation, 
and  tools  used  in  the  project  tend  to  be  harder  to  locate.  Often  people  cannot  remember  how 
tasks  were  performed  or  what  was  done  on  older  projects. 

4.2  Establish  Site  Visit  Schedule 

SEI  provides  a  strawman  site  visit  schedule  in  the  SCE  training  manual.  This  straw- 
man  has  been  elaborated  to  illustrate  the  breadth  and  depth  of  interview  coverage  desirable 
during  the  three-day  site  visit. 
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Table  3  represents  a  suggested  scheme  for  allocating  time  for  interviews  during  a 
site  visit.  The  job  titles  used  in  the  table  are  generic  and  refer  to  typical  areas  of  responsi¬ 
bility.  The  amount  of  time  allocated  to  interview  individuals  in  an  area  is  proportional  to 
the  number  of  KPAs  typically  under  the  responsibility  of  that  individual.  The  SCE  team 
should  use  organizational  charts  and  other  documentation  to  identify  the  actual  titles  and 
names  of  the  individuals  with  the  responsibilities  listed  in  Table  3.  In  addition,  it  is  recom¬ 
mended  that  the  SCE  team  lead  verify  the  actual  scope  of  responsibilities  for  each  interview 
candidate  prior  to  establishing  the  site  visit  schedule  so  the  proper  adjustments  can  be  made 
to  the  schedule. 

Some  functions  may  be  grouped  under  a  single  person,  others  may  be  spread  across 
several  individuals.  Table  4  identifies  the  typical  responsibilities  of  these  key  positions  as 
they  relate  to  the  KPAs.  The  numbers  in  Table  4  correspond  to  the  order  in  which  the  KPAs 
can  be  probed  during  the  interview.  These  priorities,  however,  are  very  dynamic.  They  will 
change  based  on  the  responses  the  SCE  team  receives  from  earlier  interviews.  If  time  runs 
out  during  the  interview,  the  KPAs  with  the  highest  priority  will  at  least  be  addressed. 

Table  3.  Allotted  Time  for  Interviews 


Position 

Length  of 
Interview  (hrs) 

Number 

Interviewed 

Project  Managers 

0.50 

1 

Software  Managers 

1.25 

3 

Manager  of  SQA 

0.50 

1 

Project  SQA 

0.50 

1 

Manager  of  SCM 

0.50 

1 

Project  SCM 

0.50 

1 

SEPG  Manager 

1.00 

1 

Test  Engineer 

0.50 

1 

System  Engineer 

0.50 

1 

Manager  of  Subcontractor 

0.50 

1 

Developer 

0.75 

2 

Using  the  allocation  from  Table  3,  the  site  visit  schedule  may  look  something  like 
Figures  4, 5,  and  6.  These  schedules  allow  15-minute  breaks  between  interviews.  This  time 


can  be  used  to  discuss  findings,  modify  an  interview  approach,  check  documentation  or 
organization  charts,  modify  the  schedule  for  the  day,  or  take  a  necessary  break.  It  is  recom¬ 
mended  that  the  SCE  team  use  the  breaks  to  gain  consensus  along  the  way  and  to  avoid 
over-scheduling.  Always  prioritize  the  list  of  people  to  interview  and  the  topics  to  be  cov¬ 
ered  during  each  interview.  Allow  extra  time  in  the  schedule  to  accommodate  unanticipated 
interviews  or  document  reviews. 


Table  4.  Topic  Priorities  for  Interviews 


Position 

Priorities  of  Key  Process  Areas  ^ 

i 

pL, 

CO 

SPTO 

SSM 

SQA 

SCM 

OPD 

CO 

Pm 

CO 

y 

Project  Managers 

■ 

■ 

2 

■ 

■ 

■ 

□ 

■ 

■ 

■ 

■ 

Software  Managers 

■ 

■ 

■ 

■ 

7 

Manager  of  SQA 

■ 

■ 

■ 

■ 

□ 

5 

[1 

3 

Project  SQA 

■ 

■ 

■ 

■ 

□ 

5 

■ 

3 

Manager  of  SCM 

■ 

■ 

■ 

4 

■ 

■ 

Project  SCM 

■ 

■ 

■ 

3 

2 

4 

■ 

■ 

■ 

SEPG 

■ 

■ 

■ 

■ 

D 

■ 

■ 

Test  Engineer 

Q 

■ 

■ 

■ 

■ 

■ 

1 

System  Engineer 

■ 

■ 

■ 

■ 

■ 

Manager  of  Subcontract 

■ 

■ 

■ 

■ 

4 

Developer  (2) 

■ 

■ 

■ 

■ 

■ 

D 

■ 

il 

L 

a.  RM  -  Requirements  Management;  SPP  -  Software  Project  Planning;  SPTO  - 
Software  Project  Tracking  and  Oversight:  SSM  -  Software  Subcontract  Manage¬ 
ment;  SQA  -  Software  Quality  Assurance;  SCM  -  Software  Configuration  Man¬ 
agement;  OPF  -  Organization  Process  Focus;  OPD  -  Organization  Process 
Definition;  T  -  Training  Program;  ISM  -  Integrated  Software  Management;  SPE  - 
Software  Product  Engineering;  IC  -  Intergroup  Coordination;  PR  -  Peer  Reviews. 

Figure  5  provides  the  sample  schedule  for  the  first  day  of  the  site  visit.  The  site  visit 
begins  with  time  allotted  for  the  SCE  team  to  pass  through  security  and  to  get  situated  in 
the  interview  room.  The  entrance  briefings  are  scheduled  to  begin  at  8:30  a.m.  for  the  eval¬ 
uation  team  and  contractor. 


The  remainder  of  the  morning  will  be  spent  reviewing  documents.  It  is  important 
for  the  SCE  team  to  do  preliminary  documentation  review  prior  to  the  interviews  so  that 
they  are  generally  acquainted  with  the  types  of  documents  the  contractor  supplied  to  sup¬ 
port  their  defined  process.  It  also  allows  the  SCE  team  to  ask  more  pointed  interview  ques¬ 
tions  based  on  what  they  learned  from  the  document  review. 

•  7:30-8:30  SCE  team  arrives  on  site 

•  8:30-9:00  SCE  team  introduction  briefing  to  contractor 

•  9:00-10:00  Contractor  entrance  briefing 

•  10:00-1 :30  Documentation  review  with  lunch 

•  1:30-5:30  Interviews 
Hrs  1.25  Software  manager  (project  1) 

0.25  Break 

0.50  Project  manager  (project  1) 

0.25  Break 

1.00  Manager  of  software  process  improvement 
0.25  Break 
0.50  SQA  manager 

•  5:30-7:30  End-of-day  caucus 

•  7 :30-f-  Document  review  at  hotel 

Figure  4.  Schedule  for  Day  One 

The  second  day  of  the  site  visit  involves  interviewing  personnel  responsible  for  spe¬ 
cific  KPAs  such  as  SCM  and  SQA.  Substantial  time  is  also  allotted  for  documentation 
review.  Refer  to  Figure  5  for  additional  details. 

The  third  day  of  the  site  visit  is  reserved  primarily  for  consolidation  interviews, 
team  caucus,  and  documenting  the  SCE  findings.  Refer  to  Figure  6  for  details. 
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*  7:30-8:30  Document  review 

♦  8:30-12:00  Interviews 

Hrs  0.50  SQA  (project  2) 

0.25  Break 

0.50  Subcontractor  software  manager  (project  2) 

0.25  Break 

0.50  System  engineer  (Project  1) 

0.25  Break 
0.50  SCM  manager 
0.25  Break 

0.50  Test  engineer  (project  2) 

•  12:00-2:00  Document  review  with  lunch 

•  2:00-5:00  Interviews 

Hrs  0.75  Developer/Group  lead  (project  2) 

0.25  Break 

1,25  Software  manager  (project2) 

0.25  Break 

0.50  SCM  (project  1) 

*  5:30-7:30  End-of-day  caucus 

Figure  5.  Schedule  for  Day  Two 

4.3  Submit  Documentation  Requests 

Organization  and  project  documentation  is  requested  at  three  stages  for  an  SCE: 
or  to  a  site  visit,  at  the  start  of  a  site  visit,  and  during  the  interviews. 


•  7:30-9:00  Document  review 

•  9:00-10:30  Team  meeting/consolidation  plan 

•  10:30-12:30  Interviews 

Hrs  0.50  Developer/Group  lead  (project  1) 

0.50  Software  manager  (project  3) 

0.25  Break 
0.75  other 

•  12:30- 1 :30  Additional  documentation  review  with  lunch 

•  1 :30-5:00  Preparation  of  findings 

Figure  6.  Schedule  for  Day  Three 

Prior  to  arrival  on  site,  the  SCE  team  may  request  a  Software  Process  Improvement 
Plan,  project  profiles,  organization  charts,  and  a  completed  SEI  questionnaire.  When  per¬ 
forming  SCEs  for  contract  monitoring  purposes,  a  software  development  plan  (SDP) 
should  also  be  requested.  The  SDP  should  be  either  generic,  a  corporate  standard,  or  from 
one  of  the  projects  offered  for  examination. 

At  the  start  of  a  site  visit,  the  contractors  will  be  directed  to  provide  documentation 
which  supports  their  responses  to  the  maturity  questionnaire.  These  references  should  be 
organized  by  each  selected  project  and  by  question  in  the  Maturity  Questionnaire.  The  SCE 
team,  however,  should  be  aware  that  the  supporting  rationale  provided  by  the  contractors 
may  not  be  accurate,  up  to  date,  or  substantiate  the  questionnaire  response.  Hence,  it  must 
be  reviewed  thoroughly.  Figure  7  lists  the  documents  typically  requested  at  the  start  of  the 
site  visit. 

During  a  site  visit  the  team  may  request  additional  documentation.  These  requests 
come  as  a  result  of  information  gained  during  the  interview  process.  Information  contained 
in  these  documents  will  be  used  to  support  information  learned  during  the  interview,  and 
may  be  used  to  corroborate  findings  in  the  final  report.  Sample  documents  to  collect  during 
the  interview  process  are  listed  in  Figure  8. 
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Project  Documents:  Program  Management  Plan 

Software  Development  Plan 
Software  Configuration  Management  Plan 
Software  Quality  Assurance  Plan 
Software  Test  Procedures 
Software  Standards  and  Procedures  Manual 
Sample  Software  Development  Folder 
Division  Documents:  Software  Policy,  Standards,  and  Procedures 

Generic  Software  Development  Plan 
Software  Quality  Assurance  Plan 
Software  Configuration  Management  Plan 

Figure  7.  Documents  To  Request  Upon  Arrival 


4.4  Assemble  Packing  List 

A  list  of  items  each  SCE  team  member  must  bring  to  a  site  visit  should  be  prepared. 
Figure  9  contains  a  sample  list. 

•  CMM  manual 

•  Spiral  notebook  (blank) 

•  KPA  interview  scripts 

•  SCE  team  members  guide  appendices 

•  Project  profiles,  questionnaire  responses,  organization  charts,  soft¬ 
ware  process  improvement  plan  (i.e.,  contractor  s  RFP  response) 

•  Blank  forms:  interview  priority  form,  document  tracking  form 

Figure  9.  Packing  List 


4.5  Develop  Entrance  Briefing 

At  the  beginning  of  the  site  visit,  the  evaluation  team  and  the  contractor  provide 
entrance  briefings  which  respectively  identify  the  scope  of  the  SCE  and  the  contractor’s 
software  process. 


Software  Project  Planning  Documents 

•  Progress  tracking  reports  or  software  sta¬ 
tus  reports 

•  Estimation  process  (size,  schedule,  cost) 

•  Risk  management  procedures  and  plans 

Software  Project  Tracking  and  Over¬ 
sight  Documents 

•  Metrics  reports  (size,  quality,  progress, 
computer  performance) 

•  Policy  for  tracking  and  reporting  project 
status 

•  Description  of  central  estimation  data¬ 
base 

Requirements  Management  Documents 

•  Requirements  change  request 

•  Requirements  document  and  traceability 
matrix 

Software  Subcontract  Management 

Documents 

» Procedures  for  selecting  and  planning 
subcontract  work 

•  Policy  and  procedures  for  monitoring 
subcontractors 

•  Subcontractor,  SCM,  and  SQA  status 
reports 

Software  Quality  Assurance  Docu¬ 
ments 

•  Project  SQA  plan 

•  SQA  policies,  procedures,  and  standards 

•  Audit  checklists,  schedules,  non-compli¬ 
ance  reports 

•  Summary  reports  to  senior  management 
(e.g.,  non-concurrence  reports) 

Software  Configuration  Management 

Documents 

•  Software  configuration  management  plan 

•  SCM  policies,  procedures,  and  standards 

•  Change  control  board  membership,  min¬ 
utes,  action  list 


Peer  Reviews  Documents 

•  Checklist  and  schedules  (design,  code, 
and  test  case) 

•  Summary  statistics  (e.g.,  type  of  errors 
found  per  life  cycle  phase) 

•  Policy  and  procedures 

•  Minutes  (design,  code,  test  case) 

Training  Program  Documents 

•  Training  policy  and  requirements 

•  Training  records 

•  Training  plan,  schedule,  curriculum 

Organization  Process  Definition  Docu¬ 
ments 

•  Organization  software  standards,  proce¬ 
dures,  policies 

•  List  of  items  in  database,  process  library 

Organization  Process  Focus  Docu¬ 
ments 

•  Software  process  improvement  plan 

•  SEPG  membership,  responsibilities, 
charter,  minutes 

•  Practices  for  submitting  revisions  to  stan¬ 
dards  and  procedures 

Software  Product  Engineering 

•  SDP 

•  Software  development  folder,  test  plans 

Intergroup  Coordination 

•  Procedures,  schedules,  plans  for  tracking 
intergroup  issues 

•  Procedures  for  controlling  critical  depen¬ 
dencies  between  groups 

•  Minutes  from  inteigroup  technical  meet¬ 
ings 

Integrated  Software  Management 

•  Templates  (SDP,  SCM,  SQA) 

•  Tailoring  guidelines 


Figure  8.  Documents  To  Request  During  Interviews 
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The  evaluation  team’s  entrance  briefing  is  given  to  the  contractors  on  the  first  morn¬ 
ing  of  the  site  visit.  The  purpose  of  the  briefing  is  to  describe  the  SCE  process  and  what  the 
contractor  can  expect  over  the  next  three  days.  A  standard  BMD  entrance  briefing  is  avail¬ 
able  in  Appendix  E.  The  briefing  should  make  a  point  of  informing  the  contractor  that  the 
SEI  questionnaire  is  used  by  the  team  only  to  become  acquainted  with  the  contractor  and 
its  processes  and  that  the  SCE  findings  will  be  based  on  information  collected  during  the 

site  visit. 

The  contractor’s  entrance  briefing  is  an  opportunity  for  the  SCE  team  to  become 
acquainted  with  the  contractor’s  organization  and  begin  to  collect  information.  The  direc¬ 
tion  the  team  provides  to  the  contractor’s  point  of  contact  will  determine  the  quality  and 
amount  of  useful  information  received  during  the  entrance  briefing.  It  is  important  that  the 
contractor’s  entrance  briefing  not  concentrate  on  findings  from  other  SCEs  or  SPAs.  A  con¬ 
tractor  could  use  this  information  to  try  to  influence  the  SCE  team.  Since  time  is  limited, 
the  SCE  team  should  ensure  the  entrance  briefing  contains  as  much  useful  information  as 
possible.  Following  is  a  list  of  topics  the  contractor  should  include  in  the  entrance  briefing: 

•  Overview  of  organization  structure  (selected  project-  and  organization-level 
groups  that  support  the  projects) 

•  Responsibilities  of  organization-level  groups  which  support  the  software-inten¬ 
sive  projects  (e.g.,  groups  responsible  for  software  process  improvement,  soft¬ 
ware  quality  assurance,  software  configuration  management,  training  program, 
development  of  software  policies  and  standards,  and  software  technology 
change  management) 

•  Overview  of  organization’s  software  development  process  (e.g.,  standards,  pol¬ 
icies,  and  procedures) 

A  significant  amount  of  information  can  be  learned  from  a  review  of  the  contrac¬ 
tor’s  organization  charts.  This  information  can  help  the  team  plan  its  strategy  for  the  inter¬ 
views  by  identifying  specific  offices  or  individuals  responsible  for  key  management  or 
technical  functions.  The  team  should  specifically  request  that  the  briefing  include  a  discus¬ 
sion  of  the  lines  and  scope  of  authority  represented  by  the  organization  charts.  This  infor¬ 
mation  indicates  the  extent  to  which  policies  and  procedures  are  institutionalized.  It  may 
also  begin  to  indicate  inhibitors  to  process  improvement. 

During  a  discussion  of  the  organization  charts,  the  team  should  identify  or  ask 
where  the  SEPG,  SCM,  SQA,  Costing,  Standards  and  Procedures,  and  Training  managers 
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are  in  the  organization  hierarchy.  These  are  critical  functions.  The  team  may  want  to  sug¬ 
gest  or  request  a  short  briefing  by  each  of  these  managers.  The  team  can  make  a  formal 
request  for  specific  information,  possibly  to  include  organizational  charts,  in  these  brief¬ 
ings,  identifying  the  roles  and  responsibilities  of  the  individual,  the  scope  and  influence  of 
his  function  on  the  projects  being  examined,  the  resources  under  his  control,  and  the  prod¬ 
ucts,  standards,  and  tools  he  provides  to  the  rest  of  the  organization. 

4.6  Notify  Contractors 

Contractors  should  be  notified  one  week  in  advance  of  the  SCE  site  visit.  The  notice 
should  be  faxed  to  the  contractor’s  designated  point  of  contact  and  receipt  verified.  The 
notice  includes  the  specific  dates  of  the  SCE,  interview  schedule,  documentation  request, 
direction  on  the  entrance  briefing,  and  a  list  of  special  supplies  the  SCE  team  will  need.  A 
sample  of  the  notification  letter  used  on  previous  SCEs  can  be  found  in  Appendix  F. 

4.7  Establish  Interview  Approach 

The  site  visit  allows  the  SCE  team  an  opportunity  to  assess  and  verify  the  software 
practices  being  used  by  the  contractor.  During  the  site  visit,  interviews  are  conducted  with 
project  personnel,  and  project  documentation  is  reviewed  to  verify  the  adequacy  of  the 
practices  being  employed  by  the  contractor.  This  section  describes  the  approach  taken  by 
the  SCE  team  to  conduct  and  document  the  interviews. 

4.7.1  SCE  Interview  Questions 

To  ensure  consistency  among  BMD  SCEs,  a  standard  list  of  KPA  questions  has 
been  developed  and  is  available  from  IDA  on  request.  During  the  course  of  the  interviews, 
these  KPA  questions  offer  the  SCE  team  a  quick  reference  to  ensure  that  all  appropriate 
goals  are  covered.  The  questions  only  serve  as  a  starting  point  for  the  SCE  team.  Follow¬ 
up  questions  are  encouraged,  based  on  the  contractor’s  response.  If  the  KPAs  are  tailored 
to  emphasize  program  issues,  the  standard  question  set  must  be  modified  accordingly. 

4.7.2  Additional  Team  Member  Roles 

The  SEl  SCE  training  program  materials  provide  guidance  on  the  roles  of  SCE  team 
members  during  a  site  visit.  This  section  includes  additional  roles  created  from  the  lessons 
learned  in  recent  SCEs.  If  possible,  roles  should  be  rotated  during  subsequent  evaluations 
to  increase  the  experience  of  each  team  member. 


a.  Door  Keeper 

This  person  has  the  responsibility  of  ensuring  that  the  interviews  proceed  unin¬ 
terrupted.  The  door  keeper  should  arrive  at  the  site  with  signs  that  indicate  “SCE 
Interviews  In  Progress.  Please  Do  Not  Disturb.”  These  signs  should  be  posted 
for  the  duration  of  the  visit  The  door  keeper  will  ensure  that  the  doors  to  the 
interview  room  remain  closed  during  the  interview  and  will  escort  an  inter¬ 
viewee  in  or  out  of  the  room. 

b.  Introducer 

This  person  will  introduce  the  team  members  to  the  contractor  and  give  any  in¬ 
troductory  remarks.  The  introducer  should  reiterate  to  the  contractor  that  all  in¬ 
formation  will  be  kept  in  strict  confidence  and  that  all  comments  made  during 
interviews  wiU  remain  non-attributed.  Appendix  G  contains  a  list  of  the  points 
that  should  be  covered  during  the  introduction  at  the  start  of  each  interview. 

c.  Document  Tracker 

This  person  wiU  keep  a  log  of  all  documents  requested  during  the  interviews. 
The  log  should  include  the  following  information:  document  number,  name  of 
interviewee,  document  name,  who  requested  it,  associated  KPA,  delivered 
(check),  reviewed  (check).  All  documents  provided  to  the  SCE  team  during  the 
visit  should  be  logged,  distributed,  and  maintained  by  the  document  tracker.  Re¬ 
fer  to  Appendix  H  for  a  sample  document  tracking  form 

The  document  number  should  be  identified  in  the  document  log  and  on  the  doc¬ 
ument  itself.  It  is  helpful  to  number  the  documents  with  a  different  series  per 
project.  For  example,  use  lOO’s  for  project  1,  200’s  for  project  2,  and  300’s  for 
project  3.  Reserve  the  400’s  for  division-level  documents.  It  is  also  helpful  for 
the  SCE  team  members  to  record  the  document  number  in  their  notes  so  that 
they  can  easily  identify  which  documents  they  requested  and  need  to  review. 

d.  Consensus  Builder 

This  individual  ensures  that  each  team  member  is  afforded  the  opportunity  to 
express  opinion  on  a  given  issue.  The  consensus  builder  will  focus  the  team  dur¬ 
ing  discussions,  possibly  by  suggesting  what  additional  information  may  help 
the  group  reach  consensus. 
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e.  Time  Keeper 

This  individual  keeps  track  of  the  time  during  the  interviews,  breaks,  and  the 
consensus  meetings  and  is  responsible  for  keeping  the  SCE  team  on  schedule. 

f.  Report  Organizer 

This  person  maintains  the  final  reports  and  all  the  supporting  documentation. 

4.7.3  Document  Interviews 

It  is  important  that  all  information  gathered  during  the  site  visit  be  documented  so 
that  it  may  later  be  used  to  support  findings.  During  the  site  visit,  SCE  team  members  notes 
serve  to  record  interviews,  stimulate  further  questions  and  documentation  requests,  and  to 
build  consensus.  Since  time  is  limited,  the  notes  must  be  recorded  efficiently.  Experience 
has  shown  that  an  interview  note  template  may  help  organize  the  note  taker’s  thoughts  and 
better  support  the  consensus-building  process. 

Figure  10  contains  a  template  which  is  convenient  for  formatting  interview  notes. 


Questions 

Name,  title  of  interviewee 

Name  of  KPA  -  Relevant  notes  during  interview 

Requested  documents 

Name  of  document,  who  requested  it,  associated  KPA 

Figure  10.  Template  for  Interview  Notes 

The  following  procedures  are  recommended: 

a.  Use  a  new  notebook  for  each  contractor. 

The  team  avoids  the  possibility  of  confusing  one  contractor’s  findings  with  an¬ 
other  and  illegally  sharing  information  among  the  competitors. 

b.  Start  a  new  page  for  each  person  interviewed. 

Clearly  identify  the  interviewee  at  the  top  of  the  page.  This  will  eliminate  any 
confusion  regarding  who  may  have  provided  what  information. 


c.  Identify  KPAs. 

Label  the  individual  pieces  of  information  provided  during  the  interview  with 
the  name  or  initials  of  the  KPA  that  it  relates  to.  By  using  the  KPA  labels,  in¬ 
formation  will  be  easy  to  find  later  during  team  discussions. 

d.  List  documents. 

List  all  the  documents  requested  during  the  interview  in  a  box  at  the  bottom  of 
the  page.  The  document  tracker  can  then  do  a  quick  check  when  requesting  in¬ 
formation  from  the  contractor  to  ensure  all  requests  are  satisfied.  It  will  also 
help  the  evaluator  to  recall  what  documents  to  review  before  establishing  KPA 
findings. 

e.  List  questions. 

List  questions  that  arise  during  the  interview  in  a  box  at  the  top  right-hand  cor¬ 
ner  of  the  page.  If  one  team  member  is  leading  the  questioning,  it  is  not  appro¬ 
priate  for  other  team  members  to  intermpt  to  ask  their  pressing  questions. 
Rather  than  miss  an  opportunity  to  pursue  an  issue,  document  the  questions  as 
they  come  to  mind  so  they  can  be  asked  at  a  more  appropriate  time. 

f.  Identify  strengths  and  weaknesses. 

Label  relevant  notes  with  a  or  to  help  identify  strengths  or  weaknesses, 
respectively.  This  will  help  when  compiling  findings  for  each  KPA. 

4.7.4  Daily  Objectives 

To  ensure  the  team  allocates  its  time  appropriately  during  the  three-day  site  visit, 
the  following  objectives  must  be  achieved  each  day: 

a.  Day  One 

The  SCE  team  should  revisit  the  schedule  following  the  first  day  of  interviews. 
Names  of  people  to  be  interviewed  may  need  to  be  added  or  removed  fi’om  the 
schedule  for  the  succeeding  days.  Time  allocations  may  need  to  be  adjusted, 
based  on  information  learned  during  the  first  day  of  interviews.  A  list  of  all  the 
documentation  requested  but  not  yet  received  by  the  team  should  be  provided 
to  the  contractor  point  of  contact  at  the  end  of  day  one. 

After  revisiting  the  schedule,  the  SCE  team  should  gather  in  the  evening  to  dis¬ 
cuss  the  results  from  the  first  day  of  interviews.  A  high-level  pass  through  each 


KPA  should  be  performed,  writing  down  initial  thoughts  and  impressions. 
These  impressions  should  indicate  where  the  contractor  appears  to  be  strong  or 
weak.  Ail  conclusions  must  be  corroborated  by  information  provided  by  either 
another  interviewee  or  by  documentation.  The  SCE  team  should  plan  how  it  will 
uncover  additional  information  to  substantiate  its  initial  conclusions. 

b.  Day  Two 

Following  the  contractor  interviews  on  day  two,  the  SCE  team  should  be  con¬ 
tinuing  to  develop  initial  impressions  and  findings  on  the  strengths  and  weak¬ 
nesses  of  the  contractor.  The  team  may  need  to  readjust  the  site  schedule  based 
on  information  gathered  on  day  two.  For  this  reason,  the  SCE  team  should  pri¬ 
oritize  the  KPAs  that  still  require  evidence  to  support  an  unacceptable  rating. 

At  the  end  of  day  two,  each  member  of  the  SCE  team  should  write  a  preliminary 
summary  of  the  strengths  and  weaknesses  for  each  assigned  KPA;  e.g.,  the  pre¬ 
liminary  findings  should  be  approximately  90%  complete.  Each  team  member 
should  record  and  distribute  other  KPA  findings  to  appropriate  team  members. 
The  following  morning,  the  team  should  review  initial  findings  to  ensure  that 
sufficient  information  will  be  gathered  during  the  remainder  of  the  visit. 

c.  Day  Three 

At  the  beginning  of  day  three,  the  SCE  team  should  review  each  KPA  individ¬ 
ually.  This  should  be  done  in  a  round-robin  fashion.  The  discussion  should  en¬ 
able  each  team  member  to  provide  feedback  on  his  or  her  understanding  of  the 
information  gathered  during  the  visit  from  interviews  and  documents.  The  team 
should  identify  areas  of  agreement  and  disagreement  within  the  team,  and  gen¬ 
erate  a  formal  list  of  questions  that  remain  to  be  asked  in  subsequent  interviews. 

All  interviews  must  be  concluded  before  noon  on  the  third  day.  This  will  allow 
sufficient  time  for  the  team  to  prepare  its  findings.  Before  the  team  departs  from 
the  contractor’s  facility  at  the  end  of  the  third  day,  the  SCE  team  should  com¬ 
plete  the  report  of  its  findings.  This  schedule  assumes  that  all  exit  briefings  of 
the  findings  occur  at  a  later  time;  refer  to  Section  5  for  further  details. 

4.7,5  Other  Interview  Practices 

Following  are  other  good  practices  for  performing  SCEs. 


a.  Record  information  from  document  review. 

During  any  phase,  details  must  be  kept  on  information  found  whde  reviewing 
documents.  The  individual  reviewing  a  document  should  log  the  title  and  date 
of  the  document  in  their  notebook.  If  a  process  or  lack  of  a  process  is  identified 
in  the  documentation,  the  facts  should  be  logged  in  the  team  member  s  note¬ 
book. 

b.  Establish  privacy  for  SCE  team. 

The  SCE  team  requires  privacy  during  the  interview  process.  The  SCE  team  co¬ 
ordinator  should  request  a  soundproof  room  with  locking  doors  if  one  is  avail¬ 
able.  A  room  with  non- soundproof  walls  will  not  permit  free  discussion  among 
the  SCE  team  members. 

c.  Use  the  CMM. 

The  members  of  an  SCE  team  should  use  the  CMM  as  reference  material  during 
a  site  visit.  This  material  contains  valuable  information  and  should  often  clarify 
any  confusion  among  team  members.  Each  team  member  should  bring  a  person¬ 
al  copy  to  each  site  visit. 

d.  Be  aware  of  terminology  differences. 

Terms  may  be  understood  by  people  to  have  different  meanings.  An  example  of 
this  is  the  term  “peer  review.”  Many  contractors  use  other  term  to  represent  the 
same  activity,  e.g.,  walk  through,  inspection,  and  desk  check.  Even  commonly 
understood  terms  may  be  used  by  a  contractor  in  an  entirely  different  way.  The 
SCE  team  must  be  flexible  in  conversing  with  a  contractor.  Understand  the  con¬ 
tractor’s  use  of  a  term  rather  than  insisting  that  the  contractor  use  the  terms  fa¬ 
miliar  to  the  team. 

e.  Conduct  interviews  with  the  entire  SCE  team  present. 

Although  time  is  limited  during  the  SCE  site  visit,  it  is  critical  that  the  team  re¬ 
main  together  and  conduct  every  interview  with  all  the  team  members  present. 
Due  to  schedule  constraints,  it  is  tempting  to  break  the  team  into  subgroups  to 
conduct  interviews  in  parallel.  When  building  consensus  the  team  will  need  in¬ 
put  from  every  member.  Thus,  it  is  vital  that  all  interviews  occur  as  a  group. 
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f.  Take  advantage  of  breaks  between  interviews. 

The  breaks  between  interviews  can  be  effectively  used  by  the  SCE  team.  Breaks 
provide  an  opportunity  for  the  team  members  to  discuss  information  gathered 
during  the  last  interview  and  information  remaining  to  be  confirmed  or  pursued 
during  the  next  interview.  Breaks  can  also  be  used  by  the  team  members  to  dis¬ 
cuss  the  interview  process,  or  a  team  member’s  approach  to  seeking  specific  in¬ 
formation,  or  to  clarify  or  improve  the  effectiveness  of  questions  being  asked. 

g.  Establish  consensus. 

During  the  consensus  building  process  the  team  should  listen  to  each  member. 
The  Consensus  Builder  is  responsible  for  ensuring  that  this  occurs.  Avoid  trying 
to  force  consensus  too  early  since  information  is  still  being  collected.  Verify  all 
conclusions  by  cross-checking  with  other  team  members’  interview  notes  and 
with  documents  provided  by  the  contractor. 

It  is  important  to  avoid  having  one  person  dominate  the  process  and  rule  over 
the  SCE  team.  It  is  best  if  everyone’s  opinions  and  concerns  are  heard  and  ad¬ 
dressed.  Hence  it  is  important  to  make  the  environment  conducive  for  discus¬ 
sion  among  all  team  members,  not  just  the  most  vocal  members.  If  possible,  the 
team  leader  and  the  consensus  builder  should  attend  a  course  on  facilitation. 

It  is  important  to  determine  early  if  there  are  different  opinions  on  whether  the 
contractor  is  satisfying  a  KPA.  If  consensus  is  not  established,  this  indicates  that 
more  information  is  needed.  Identify  what  information  is  necessary  to  support 
the  different  view  points  and  appropriately  adjust  the  interview  schedule  and 
document  request  list.  Consensus  must  be  established  when  ranking  the  KPAs 
as  acceptable  or  unacceptable.  It  is  less  critical,  however,  to  have  consensus  on 
the  contractor’s  individual  strengths  and  weaknesses  for  particular  KPAs.  In 
this  event,  the  team  should  reach  a  compromise  on  the  wording  of  the  strength 
or  weakness  to  accurately  reflect  the  contractor’s  process. 


5.  ACTIVITIES  AFTER  CONDUCTING  AN  EVALUATION 


This  section  of  the  report  will  review  the  activities  that  occur  after  the  SCE  site  visit. 
Included  in  this  section  is  a  description  of  the  final  SCE  report  and  the  feedback  to  the  con¬ 
tractor,  the  SSEB,  the  program  office,  and  BMDO  concerning  the  results  of  the  SCE. 

5.1  Final  Report 

The  SCE  team  must  prepare  a  report  of  its  findings  and  their  rationale  for  the  con¬ 
tractor,  SSEB,  program  office,  and  BMDO.  Refer  to  Figure  1 1  for  a  depiction  of  the  outline 
of  the  report. 

1.  Overview  of  Evaluation  Process 

•  Overview  of  CMM 

•  Overview  of  SCE  process 

2.  Summary  of  Findings 

•  Acceptable  KPAs 

•  Unacceptable  KPAs 

•  Summary  of  contractor’s  software  process  improvement  plans 

3.  Summary  of  Project  Information 

•  List  of  original  projects  submitted  (7-9  projects) 

•  List  of  projects  selected  (1-3  projects) 

•  Ration^e  for  project  selection 

•  Interview  schedule,  interviewee  names  and  positions 

4.  Details  of  KPAs 

•  Exit  briefing  slides 

Appendix 

Figure  11.  Report  Outline 

The  first  chapter  of  the  SCE  report  should  include  a  high-level  overview  of  the 
CMM  and  the  SCE  process  since  few  members  of  a  program  office  or  SSEB  are  familiar 
with  them.  It  is  important  to  describe  the  model  used  to  measure  the  contractors  and  the  pro- 


cess  the  SCE  team  used  to  generate  its  findings.  This  description  should  be  included  in  the 
presentation  to  the  SSEB  and  program  management  as  well. 

The  second  chapter  of  the  report  should  include  a  summary  of  the  findings.  A  list 
of  the  acceptable  and  unacceptable  KPAs  should  be  provided  along  with  an  overview  of  the 
contractor’s  software  process  improvement  plans. 

The  third  chapter  should  describe  the  focus  of  the  site  visit.  It  should  include  a  sum¬ 
mary  of  the  project  profiles  submitted  by  the  contractors  and  the  rationale  for  selecting  the 
three  projects  which  were  reviewed  by  the  SCE  team.  In  addition,  the  site  visit  schedule 
should  be  included,  identifying  the  names  and  titles  of  individuals  interviewed.  Other  items 
to  add  in  this  section  of  the  report  include  copies  of  the  contractor’s  entrance  briefing  or  the 
most  important  slides  that  would  be  meaningful  for  the  SSEB. 

The  fourth  chapter  of  the  report  will  contain  the  SCE  exit  briefing.  Each  of  the  slides 
should  contain  a  summary  statement  of  the  KPA,  and  a  list  of  KPA  strengths,  weaknesses, 
and  noted  improvement  activities  the  contractor  has  planned  or  is  underway.  The  summary 
statement  of  the  KPA  should  identify  whether  the  KPA  was  acceptable  or  unacceptable  and 
the  prevailing  rationale.  This  is  the  most  important  section  of  the  report.  Refer  to  Figure  12 
for  an  example  of  an  exit  briefing  slide. 

The  SCE  team  should  make  an  effort  to  identify  strengths  and  weaknesses  for  all 
KPAs.  For  example,  there  may  be  aspects  of  the  contractor’s  process  that  are  viewed  as 
weaknesses  even  though  the  KPA  is  rated  as  acceptable  overall.  The  noted  weaknesses  are 
very  informative  to  the  program  manager  and  the  SSA  since  they  help  to  identify  potential 
risks.  Avoid  having  no  strengths  or  weaknesses  noted  for  any  KPA. 

In  this  report.  Appendix  I  contains  a  list  of  strengths  and  weaknesses  for  each  of  the 
KPAs  used  by  previous  BMD  teams.  The  SCE  team  can  use  these  to  help  formulate  the 
wording  of  its  findings.  It  is  important,  however,  to  use  only  the  findings  that  have  been  ver¬ 
ified  through  the  interview  process  or  documentation  reviews. 

The  appendix  of  the  final  report  should  contain  the  team  members’  notebooks,  the 
contractor’s  project  profiles,  and  responses  to  the  maturity  questionnaire.  The  team  mem¬ 
bers’  notebooks  contain  documentation  of  the  interviews  and  document  reviews  conducted 
at  the  contractor’s  facility. 

The  report  must  be  labeled  Proprietary  Information.  In  support  of  the  source  selec¬ 
tion  process,  the  report  must  also  be  labeled  as  Source  Selection  Sensitive.  In  this  case,  the 
report  should  be  made  avaUable  only  to  those  that  have  signed  the  proper  Procurement 


Peer  Reviews 

SuiTiin3ry:  Peer  Review  process  is  unacceptable:  the  process  is  informal 

and  not  applied  consistently  across  all  projects. 

Strengths: 

•  Some  projects  perform  low-level  inspections  of  critical  modules 
Weaknesses: 

•  Design,  code,  unit  test  cases  not  consistently  reviewed  across  projects 

•  No  formal  procedures  for  conducting  peer  reviews  (e.g. ,  checklist) 

•  Lack  of  formal  reporting  and  tracking  of  peer  review  findings 

•  Lack  of  statistics  on  findings,  results,  and  effort 

Planned  Improvement  Activities 

•  None 

Figure  12.  Sample  Exit  Briefing  Slide 

Integrity  forms,  e.g.,  the  SSEB,  the  element  program  manager,  element  software  lead,  and 
the  Director  of  Computer  Resources  Engineering  at  BMDO. 

5.2  Contractor  Feedback  of  Evaluation  Results 

The  contractor  may  provide  feedback  on  the  SCE.  When  SCEs  are  used  for  source 
selection,  the  competing  contractors  will  not  receive  an  exit  briefing  at  the  end  of  the  SCE 
site  visit  Instead,  the  winning  contractors  will  be  briefed  soon  after  the  contract  is  awarded. 
At  that  time  the  winning  contractors  have  an  opportunity  to  provide  feedback  of  the  SCE 
results  and  to  clarify  any  questions.  The  losing  contractors  will  receive  a  very  high-level 
overview  of  the  SCE  results  at  the  “loser’s  conference,”  and  no  detailed  feedback  will  be 
provided  of  the  SCE  results  at  that  time. 

When  SCEs  are  used  to  help  monitor  a  contract,  the  contractors  will  receive  a 
detailed  exit  briefing  at  the  end  of  the  SCE  site  visit.  The  exit  briefing  offers  the  contractor 
an  opportunity  to  understand  the  SCE  results  and  to  question  the  team  on  its  findings.  Since 
the  exit  briefing  slides  are  not  very  detailed  and  the  audience  is  generally  very  large,  con¬ 
tractors  may  request  a  follow-up  meeting  with  their  SEPG  and  the  SCE  team  to  ensure  that 


the  findings  are  interpreted  correctly  and  incorporated  into  their  software  process  improve¬ 
ment  plans. 

5.3  Use  of  Evaluation  Results  in  Source  Selection 

One  of  the  most  important  program  management  responsibilities  in  using  an  SCE 
for  source  selection  is  defining  how  the  SCE  results  should  be  communicated  to  the  SSEB. 
The  SCE  team  and  the  SSEB  must  agree  upon  the  format.  One  successful  method  is  for  the 
SCE  team  to  provide  the  SSEB  a  copy  of  the  SCE  report  after  each  individual  site  visit  or 
after  all  site  visits  have  been  completed.  The  report  contains  a  summary  of  the  contractor’s 
strengths,  weaknesses,  and  improvement  plans  for  each  KPA.  In  addition,  the  summary 
identifies  each  KPA  as  acceptable  or  unacceptable.  Refer  to  Section  5.1  for  additional 
details  of  the  final  report  format. 

This  summary  of  the  KPAs  is  used  by  the  SSEB  and  is  incorporated  with  the  rest  of 
the  source  selection  process.  Refer  to  Section  3.2  for  details  of  the  criteria  the  SSEB  may 
use  to  assign  a  color,  numerical,  or  risk  rating.  In  the  event  that  the  SSEB  requires  addition¬ 
al  clarification  of  the  findings,  the  SCE  team  leader  should  be  available  to  respond  to  any 
questions. 

5.4  Use  of  Evaluation  Results  for  Contract  Monitoring 

When  an  SCE  is  performed  to  help  monitor  a  contract,  the  SCE  team,  with  the 
approval  of  the  program  manager,  should  present  its  findings  on  the  morning  of  the  fourth 
day  to  the  contractor.  The  three-day  SCE  schedule  is  too  full;  there  is  no  time  to  present  the 
findings  any  earlier.  As  part  of  the  contract  monitoring  process,  the  program  office  should 
compare  the  results  of  the  evaluation  with  the  contractor’s  process  improvement  plans.  If 
there  is  a  discrepancy  between  the  weaknesses  identified  by  the  independent  SCE  team  and 
those  of  the  contractor’s  self-assessment,  the  contractor  should  be  made  aware  of  the  dif¬ 
ferences.  The  program  office  should  ensure  that  the  contractor  has  an  acceptable  plan  to 
mitigate  the  risks  that  could  result  firom  the  weaknesses  and  to  reduce  the  principal  weak¬ 
nesses  over  the  length  of  the  contract 

There  are  several  approaches  the  program  office  can  take  to  encourage  contractors 
to  improve  their  software  development  processes.  One  is  to  emphasize  to  the  contractor  the 
importance  of  software  process  improvement  in  the  BMD  program.  If  there  are  two  or  more 
contractors  competing  for  a  future  contract  the  program  office  can  state  its  plans  to  use 
SCE  results  during  the  next  source  selection  process.  This  should  help  to  motivate  the  com¬ 
peting  contractors  to  improve  so  that  they  can  better  their  scores  during  the  next  source 
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selection.  In  addition,  the  contractors  can  report  the  status  of  their  process  improvement 
program  at  the  software  reviews  (e.g.,  preliminary  design  review  or  critical  design  review). 
At  the  program  reviews,  the  contractors  should  be  required  to  describe  their  improvement 
plans  and  accomplishments. 

Another  means  of  encouraging  contractors  to  improve  is  to  incorporate  software 
process  improvement  into  an  award-fee  program.  Since  process  improvement  focuses  on 
the  organization  rather  than  on  a  specific  program,  not  all  projects  are  good  candidates  for 
an  award  fee.  The  best  candidates  are  those  contractors  who  have  an  organization  devoted 
to  the  BMD  program  and  are  working  on  several  BMD  projects.  For  example,  the  National 
Test  Bed  and  Integration  Contractor  (NTBIC)  would  be  a  suitable  candidate. 

If  an  award  fee  is  suitable,  the  award  fee  plan  should  contain  incentives  for  improv¬ 
ing  the  software  process.  Depending  on  the  length  of  the  contract  and  the  initial  maturity 
of  the  contractor,  the  plan  should  include  intermediate  goals  that  emphasize  the  benefits  of 
advancing  from  a  level  1  maturity  up  to  a  level  5.  Since  it  is  estimated  to  take  at  least  a  year 
to  improve  from  one  level  to  another,  the  award  fee  should  be  staggered.  Assuming  that  the 
contractor’s  initial  maturity  rating  is  a  level  1,  there  should  be  a  minimal  award  at  the  end 
of  the  first  year  for  advancing  from  a  level  1  to  a  level  2.  At  the  end  of  the  second  or  third 
year,  there  should  be  a  larger  reward  for  advancing  from  a  level  2  to  a  level  3,  and  so  on. 
Sample  award  fee  plans  can  be  provided  by  BMDO. 

5.5  Registry  of  Evaluation  Results 

As  described  in  Sections  3, 4,  and  5,  the  preparation,  conduct,  and  follow-up  for  an 
SCE  consume  considerable  amounts  of  time,  effort,  and  expense  on  the  part  of  both  the 
Government  and  the  contractor.  Subjecting  a  contractor  to  multiple  SCEs  in  the  context  of 
several  procurements  compounds  this  problem.  Reducing  the  results  of  an  SCE  to  a  mean¬ 
ingful,  concise  form  that  can  be  stored  in  some  repository  and  reused  in  later  procurements 
can  greatly  reduce  this  time,  effort,  and  expense.  Use  of  an  SCE  repository  could  also  short¬ 
en  the  procurement  cycle  considerably  by  reducing  the  number  of  “fresh”  SCEs  that  must 
be  done  in  order  to  evaluate  fairly  and  fully  the  software  development  process  maturity  of 
all  the  contractors  who  may  respond.  Therefore,  SCE  results  should  be  archived  by  BMDO 
for  potential  use  in  future  procurements. 

5.6  Issues  Related  to  SCE  Findings 

Based  on  the  earlier  experiences  of  BMD  SCE  teams,  there  are  several  issues  asso¬ 
ciated  with  the  findings  that  should  be  clarified  for  future  evaluations:  consistency,  evalua- 
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tion  of  new  divisions,  rating  method,  weighting  the  results,  maturity  of  the  process,  and 
saving  evidence.  Each  issue  is  discussed  in  more  detail  in  the  following  subsections. 

5.6.1  Establish  Consistency  Between  Site  Visits 

When  performing  several  SCEs  in  support  of  a  source  selection,  how  consistent  do 
the  findings  have  to  be?  It  is  important  to  probe  each  of  the  offerors  the  same  way  initially 
for  each  KPA.  However,  after  the  first  pass  of  general  questioning,  the  SCE  team  will  ask 
follow-up  questions  which  are  based  on  the  offeror’s  initial  responses.  The  follow-up  ques¬ 
tions  do  not  have  to  be  identical  from  one  offeror  to  the  next. 

Similarly,  the  findings  do  not  have  to  be  identical.  It  is  not  important  to  have  the 
same  number  of  strengths  and  weaknesses  with  the  same  wording  for  each  offeror  unless 
they  have  identical  processes.  It  is  more  important  for  the  SCE  team  to  discriminate 
between  offerors. 

5.6.2  Evaluate  New  Divisions 

Many  defense  contractors  are  routinely  reorganizing  and  consolidating  divisions. 
This  may  make  it  difficult  for  an  SCE  team  to  evaluate  an  offeror’s  standard  software  devel¬ 
opment  process  which  is  in  the  works.  When  two  divisions  are  in  the  process  of  merging 
their  software  development  policies,  procedures,  and  practices,  the  SCE  team  must  look  at 
both  divisions.  The  SCE  findings  should,  in  essence,  reflect  the  least  common  denominator. 
In  other  words,  if  one  of  the  former  divisions  has  a  well-established  process  but  the  other 
one  does  not,  this  must  be  reflected  in  the  findings  as  a  weakness.  It  is  also  important  to 
understand  how  the  new  policies  are  being  adopted  and  enforced  by  both  of  the  pre-existing 
divisions. 

5.6.3  Color  Code  KPA  Summary  Statements 

In  the  past,  each  KPA  summary  statement  characterized  whether  the  KPA  is  evalu¬ 
ated  by  the  SCE  team  as  acceptable  or  unacceptable.  This,  however,  may  not  offer  a  pro¬ 
gram  manager  or  SSEB  sufficient  discrimination  between  offerors.  It  is  better  if  the  SCE 
team  can  identify  within  the  summary  statement  whether  the  individual  KPA  is  blue,  green, 
yellow,  or  red.  Previous  SCEs  used  several  qualitative  words  to  distinguish  between  these 
categories.  Refer  to  Figure  13  for  details. 
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Blue  Worils 

Green  Words 

Exceptional 

Adequate 

Superior 

Acceptable 

Complete 

Sufficient 

Outstanding 

Yellow  Words 

Red  Words 

Inadequate 

Unacceptable 

Insufficient 

Scarce 

Incomplete 

Flawed 

Impaired 

Significantly  deficient 

Figure  13.  Qualitative  Words  for  Use  in 
KPA  Summary  Statements 


It  is  important  to  coordinate  with  the  SSEB  in  advance  of  the  SCE  in  order  to  define 
the  qualitative  terms  that  will  be  the  most  meaningful. 

5.6.4  Weight  SCE  Results 

In  the  past,  SCE  teams  evaluated  both  the  subcontractors  and  the  prime  contractors 
responsible  for  developing  or  integrating  any  portion  of  the  software  system.  The  SCEs 
were  performed  on  these  two  types  of  contractors  regardless  of  the  amount  of  software  they 
each  developed.  A  better  approach  is  to  determine  if  the  contractor  is  suitable  for  an  SCE 
once  the  SSA  is  informed  of  the  division  of  labor.  If,  for  example,  a  prime  contractor  is 
developing  90%  of  the  software  and  its  subcontractor  only  10%,  it  may  be  cost  effective  to 
only  evaluate  the  prime  contractor.  If  the  subcontractor  is  developing  90%  of  the  software, 
the  SCE  should  be  done  on  only  the  subcontractor.  But  if  the  work  load  is  evenly  distribut¬ 
ed,  it  would  be  wise  to  evaluate  both  the  subcontractor  and  the  prime  contractor.  After  the 
SCEs  are  complete,  the  SSEB  will  weight  the  SCE  results  accordingly.  Hence  it  is  not  wise 
to  spend  the  time  and  money  performing  SCEs  on  both  types  of  contractors  if  the  results 
will  not  be  used.  This  issue  should  be  discussed  in  detail  with  the  SSA. 

5.6.5  Evaluate  Process  Maturity 

How  mature  does  a  process  have  to  be?  Frequently  an  SCE  team  will  be  exposed  to 
a  new  process  that  is  being  adopted  by  a  contractor’s  organization,  and  the  evaluation  team 


must  determine  if  it  has  been  in  place  long  enough  to  receive  an  acceptable  rating.  If  a  new 
process  is  supported  by  the  appropriate  policies  and  procedures  and  if  it  is  implemented  on 
all  new  programs,  it  may  be  evaluated  as  acceptable.  If,  however,  it  is  not  routinely  imple¬ 
mented  by  new  programs,  it  may  not  be  sufficiently  mature.  A  process  that  is  supported  by 
draft  policy  or  one  that  is  in  place  less  than  three  months  may  also  not  be  mature  enough  to 
receive  an  acceptable  rating.  However,  these  are  both  strengths  that  should  be  noted  in  the 
findings.  In  general,  the  SCE  team  must  see  evidence  that  the  process  is  institutionalized 
and  reinforced  through  policy,  training,  and  appropriate  review  processes. 

5.6.6  Save  Evidence 

The  SCE  team  member  notebooks  should  be  collected  at  the  end  of  each  site  visit 
and  given  to  the  SSA  at  the  time  of  the  final  report.  Included  with  the  notebooks  should  be 
other  pertinent  evidence,  e.g.,  actual  interview  schedule,  document  tracking  forms,  inter¬ 
view  priority  forms,  project  profiles,  questionnaires,  organization  charts,  process  improve¬ 
ment  plan,  contractor’s  entrance  briefing,  and  any  SCE  team  correspondence  with  the 
offerors  in  advance  of  the  site  visit. 


APPENDIX  A. 
RFP  WORDING 
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The  following  sample  text  illustrates  how  SCEs  might  be  inserted  within  Sections 
L  and  M  of  the  RFP.  This  example  assumes  that  two  SCE  teams  will  be  used  and  the  results 
will  be  a  risk  factor  for  source  selection.  The  outline  of  the  project  profile  and  the  summary 
recording  form  can  be  found  in  Appendix  B  and  Appendix  C,  respectively,  of  this  report. 

Section  L:  Contractor  Software  Capability  Evaluations  [example  section  L-6.3.3] 

The  Government  will  send  two  small  (three  to  five  members)  teams  of  software 
experts  to  the  appropriate  offeror  site(s)  for  three  consecutive  days  to  conduct  a  Software 
Capability  Evaluation  (SCE)  of  the  offeror’s  organization  and  processes.  One  team  will 
conduct  the  evaluation  for  the  major  software  developer  for  this  acquisition.  The  other  team 
will  conduct  the  evaluation  for  the  software  integrator  for  this  acquisition.  In  the  event  that 
the  same  contractor/site  is  doing  both  the  major  portion  of  the  development  as  well  as  inte¬ 
grating  the  software,  the  second  team  will  conduct  the  evaluation  of  the  next  largest  soft¬ 
ware  developer.  The  offeror  shall  provide  the  inputs  for  the  software  programs/projects  to 
be  reviewed  by  the  SCE  on-site  teams,  but  the  teams  will  not  review  Code-word  classified 
information  or  Sensitive  Compartmented  Intelligence  (SCI)  information  as  part  of  the 
review.  Offerors  are  cautioned  that  they  are  responsible  for  making  available  sufficient  non- 
Code-Word/non-SCI  information  for  the  Government  evaluation  team  to  conduct  a  thor¬ 
ough  evaluation.  Failure  to  do  so  may  contribute  to  an  unfavorable  evaluation.  Documen¬ 
tation  and  information  required  to  support  the  site  visit  are  identified  in  paragraph  L- 
6.43.1 .  Offerors  are  cautioned  that  the  Government  will  use  data  provided  for  the  visit  and 
information  obtained  during  the  visit  in  performing  part  of  the  source  selection  evaluation 
as  indicated  in  Section  M.  There  will  be  no  outbrief  to  the  offeror  at  the  conclusion  of  the 
SCE  site  visits. 

Appendix  E,  Software  Capability  Evaluation  (SCE)/Capability  Maturity  Model 
(CMM)  [example  section  L-6.4.3.1] 

Submit  the  information  identified  below  for  each  of  the  two  proposing  sites  (refer¬ 
ence  L-63.3  above).  Information  submitted  must  be  unclassified.  Offerors  are  advised  that 
the  Government  will  use  data  provided  by  each  offeror  in  this  Appendix,  Appendix  G,  on 
past  performance,  and  information  obtained  through  the  SCE  site  visit  in  performing  the 
evaluation  in  this  area.  If  offerors  submit  information  relative  to  Code-word  classified  pro¬ 
grams,  they  are  cautioned  that  the  evaluators  will  not  review  information  or  material  clas¬ 
sified  higher  than  Secret  Offerors  remain  responsible  for  providing  the  on-site  evaluators 
with  accessible  information  that  demonstrates  the  offeror’s  software  capability. 
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a.  Current  Capability.  The  offeror(s)  will  complete  the  Software  Engineering 
Institute  (SEI)  Software  Process  Maturity  Questionnaire  (MQ)  (version  1.1, 
dated  April  1994)  for  three  to  five  (3  to  5)  current  projects  at  each  of  the  two 
sites  (reference  L-6.3.3  above).  (A  project  that  has  been  completed  is  acceptable 
only  if  it  has  been  completed  after  January  31, 1994.)  The  offeror  should  select 
those  projects  that  best  match  the  engineering  requirements  of  this  contract.  For 
offerors  with  fewer  than  three  current  projects  at  the  proposing  site,  submit  MQ 
responses  for  as  many  projects  as  are  available.  For  each  “yes”  response,  please 
note  on  a  separate  comment  sheet  the  mechanism  or  documentation  to  justify 
the  response.  The  MQ  can  be  found  in  Attachment  5  to  these  instructions.  The 
answers  to  the  questions  will  be  submitted  in  this  Appendix.  There  is  no  specific 
page  limit  for  completing  the  questionnaire. 

b.  Project  Profile.  For  each  project,  the  offeror(s)  will  complete  a  Project  Profile 
and  attach  it  to  the  respective  response  to  the  Maturity  Questionnaire,  and  sub¬ 
mit  it  in  this  Appendix.  The  Project  Profile  Outline  can  also  be  found  in  Attach¬ 
ment  6  to  these  instructions.  Limit  Project  Profiles  to  one  (1)  page  per  project. 
The  organization  charts  required  by  the  last  instruction  in  Attachment  6  do  not 
count  in  the  page  limits. 

c.  Software  Process  Improvement  Plan  (SPIP).  Submit  a  synopsis  of  the  proposing 
sites’  SPIP  in  offeror  format  in  this  Appendix.  This  synopsis  shall  be  limited  to 
25  pages.  The  SPIP  should  communicate  the  offerors’  current  software  process 
capability  as  well  as  the  desired  maturity  level,  specific  planned  improvements, 
dedicated  resources,  effort  estimates,  and  a  time  phasing  of  those  improvements 
to  bring  the  offerors’  software  capability  to  the  desired  maturity  level.  Ensure 
that  significant  planned  improvements  are  reflected  in  the  Integrated  Master 
Plan  (paragraph L-6 .4  J 2. c). 

d.  Points  of  Contact.  Provide  for  each  proposing  site  a  point  of  contact  and  an 
alternate  point  of  contact  to  schedule  the  visit  and  arrange  all  the  administrative 
aspects.  Provide  their  names,  location,  phone  numbers,  fax  numbers,  and 
address.  If  security  authorization  is  necessary  for  the  SCE  team  to  be  at  the  off¬ 
erors’  facilities,  the  name,  fax  number,  and  telephone  number  of  the  security 
officer(s)  should  also  be  provided  along  with  any  other  pertinent  information 
required  to  obtain  security  approval. 
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Management  Area  [example  section  M3.2] 

This  area  is  divided  into  three  items.  Items  M-1  and  M-2  are  equal  in  importance 
and  are  more  important  than  item  M-3.  For  Item  M-3,  the  evaluation  will  include  the 
Appendix  and  the  results  of  the  on-site  Software  Capability  Evaluation  (SCE)  Team  review. 
For  the  SCE,  the  Government  shall  not  consider  Code-word  classified  or  Sensitive  Com- 
partmented  Intelligence  (SCI)  information. 

Item  M-3;  Past  Performance,  Past  Experience,  and  Software  Capability.  These  three 
elements  will  be  given  equal  consideration  and  an  evaluation  of  Item  M-3  will  be  done  at 
the  item  level.  This  item  will  be  assessed  based  on 

1 .  Record  of  previous  performance  (quality,  cost  effectiveness,  timeliness  and  cus¬ 
tomer  satisfaction). 

2.  Range  (varied  contracts,  number  of  contracts)  and  significance  (size,  complex¬ 
ity,  participation)  of  relevant  corporate  experience  (including  subcontractor 
experience)  in  accomplishing  the  efforts  described  in  the  SOW. 

3.  Software  Capability  Evaluation  (SCE).  The  offeror’s  software  capability  as 
evaluated  by  the  government  SCE  team  based  upon  (a)  the  SEI  Software  Pro¬ 
cess  Maturity  Questionnaire,  (b)  Project  Profile,  (c)  Software  Process  Improve¬ 
ment  Plan,  and  (d)  the  site  visit. 


APPENDIX  B. 

PROJECT  PROFILE  OUTLINE 


# 


The  following  form  is  a  sample  project  profile.  A  project  profile  should  be  filled  out 
for  each  project. 

a.  Project  Name:  name  of  project  listed  on  the  contract. 

b.  Project  Number:  unique  identifying  number  on  the  contract. 

c.  Project  Type:  e.g.,  scientific,  human-machine,  business,  control,  support  soft¬ 
ware. 

d.  Customer:  the  agency  that  procured  the  software  and  a  point  of  contact  within 
that  organization. 

e.  Subcontractors/Prime  Contractors:  list  any  subcontractors  employed  on  the 
project  or  list  the  prime  contractor  if  the  offeror  was  a  subcontractor. 

f.  Current  Phase:  identify  the  current  phase  of  the  software  development  pro¬ 
cess,  e.g.,  requirements  definition,  detailed  design,  code  and  unit  test,  integra¬ 
tion  test,  maintenance. 

g.  Location:  primary  site  and  organizational  unit  (e.g.,  division  name)  responsible 
for  the  software  development  effort. 

h.  Start  Date:  starting  date  of  the  contract. 

i.  Design  Completion  Date:  estimated  or  actual. 

j.  Code  Completion  Date:  estimated  or  actual. 

k.  End  Date:  contract  completion  date. 

l.  Estimated  Team  Size:  peak  staff-month  period  and  average  staff-years  over  the 
contract  period. 

m.  Estimated  KSLOC:  estimated/actual  thousand  source  lines  of  code. 

n.  Programming  Languages:  percentage  of  KSLOC  in  languages  (e.g.,  Ada,  For¬ 
tran,  Pascal,  C,  Assembler). 

o.  Target  Hardware  System:  computer  on  which  software  executes. 

p.  Development  Hardware  System:  host  computer  for  the  compiler  and  support 
environment. 


B-3 


Applicable  Standards:  e.g.,  name  of  commercial  standards,  DOD-STD- 
2 167  A,  MIL-STD-498. 

Development  Approach:  e.g.,  rapid  prototyping,  full-scale  development,  com¬ 
mercial-off-the-shelf  (COTS)  products  integration. 

SEI  Questionnaire:  the  attached  questionnaire  and  its  answer  sheet  should  be 
completed  for  each  of  the  projects. 

Organization  Chart:  both  project-level  and  organizational-level  charts  are 
requested.  For  each  project,  provide  the  most  recent  organization  chart  with 
titles  and  individual  names.  This  chart  should  explicitly  identify  the  individu- 
al(s)  responsible  for  the  following  activities:  project  management,  system  engi¬ 
neering,  software  project  management,  software  engineering,  software  quality 
assurance,  software  configuration  management,  technical  subcontractor  man¬ 
agement,  simulation,  and  integration  and  testing.  Other  technical  software 
activities  that  should  be  identified  at  an  organizational  level  include  titles  and 
individual  names  of  persons  responsible  for  software  process  improvement, 
training  program,  evaluation  of  advanced  software  technologies,  as  well  as  the 
organization  managers  of  software  quality  assurance  and  software  configuration 
management 


APPENDIX  C. 

SUMMARY  RECORDING  FORM 
FOR  CMM  QUESTIONNAIRE 


This  document  contains  questions  about  the  implementation  of  important  software  practices  in 
your  software  organization.  The  questionnaire  should  be  completed  for  four  to  six  software 
projects  from  the  same  organizational  site  proposing  on  this  solicitation  and  for  projects  that  best 
match  the  engineering  requirements  of  the  contract,  e.g.,  size,  domain,  development  process. 
Code-word  classified  programs  or  projects  that  have  been  completed  for  more  than  four  months 
should  not  be  submitted.  Identify  the  project  names  that  correspond  to  the  assigned  labels  (A-F). 

The  questions  are  organized  in  groups  of  key  process  areas  such  as  software  requirements  man¬ 
agement  and  software  project  planning.  To  the  right  of  each  question  are  boxes  to  record  the 
response  for  each  of  the  four  to  six  projects.  There  are  four  appropriate  responses  for  each  ques¬ 
tion:  Yes  (Y),  No  (N),  Not  Applicable  (N/A),  and  Don’t  Know  (DyTC). 

1.  the  practice  is  well  established  and  consistently  performed.  The  practice  should 
be  performed  nearly  always  in  order  to  be  considered  well-established  and  consis¬ 
tently  performed  as  a  standard  operating  procedure. 

2.  Noi  the  practice  is  not  well  established  or  inconsistently  performed.  The  practice  may 
be  performed  sometimes,  or  even  frequently,  but  it  is  omitted  under  difficult  circum¬ 
stances. 

3.  Not  Applicable:  you  have  the  required  knowledge  about  your  project  or  organization 
and  the  question  asked,  but  you  feel  the  question  does  not  apply  for  your  project.  For 
example,  the  entire  section  on  “Software  Subcontract  Management”  may  not  apply  to 
your  project  if  you  do  not  work  with  any  subcontractors. 

4.  Don’t  Know:  you  are  uncertain  about  how  to  answer  the  question. 

Please  answer  all  of  the  questions.  For  each  “Yes”  response,  please  note  on  a  separate  comment 
sheet  the  mechanism  or  documentation  used  to  justify  the  response.  There  is  no  specific  page  limit 
for  completing  the  questionnaire. 

For  a  definition  of  terms,  refer  to  the  Software  Engineering  Institute’s  Software  Process  Maturity 
Questionnaire,  version  1.1.0,  April  1994,  which  is  in  the  BMDO  library. 


Project  Names 


Project  A: 
Project  B: 
Project  C: 
Project  D: 
Project  E: 
Project  F: 


QUESTIONS 


Requirements  Management 

1.  Are  system  requirements  allocated  to  software 
used  to  establish  a  baseline  for  software  engineering 
and  management  use? 

2.  As  the  systems  requirements  allocated  to  software 
change,  are  the  necessary  adjustments  to  software 
plans,  work  products,  and  activities  made? 

3.  Does  the  project  follow  a  written  organizational 
policy  for  managing  the  system  requirements  allo¬ 
cated  to  software? 

4.  Are  the  people  in  the  project  who  are  charged  with 
managing  the  allocated  requirements  trained  in  the 
procedures  for  managing  allocated  requirements? 

5.  Are  measurements  used  to  determine  the  status  of 
the  activities  performed  for  managing  the  allocated 
requirements  (e.g.,  total  number  of  requirements 
changes  that  are  proposed,  open,  approved,  and 
incorporated  into  the  baseline)? 

6.  Are  the  activities  for  managing  allocated  require¬ 
ments  on  the  project  subjected  to  SQA  review? 

Software  Project  Planning 

1.  Are  estimates  (e.g.,  size,  cost,  and  schedule)  docu¬ 
mented  for  use  in  planning  and  tracking  the  software 
project? 

2.  Do  the  software  plans  document  the  activities  to 
be  performed  and  the  commitments  made  for  the 
software  project? 

3.  Do  all  affected  groups  and  individuals  agree  to 
their  commitments  related  to  the  software  project? 

4.  Does  the  project  follow  a  written  organizational 
policy  for  planning  a  software  project? 

5.  Are  adequate  resources  provided  for  planning  the 
software  project  (e.g.,  funding  experienced  individu¬ 
als)? 


Project 


QUESTIONS 


6.  Are  measurements  used  to  determine  the  status  of 
the  activities  for  planning  the  software  project  (e.g., 
completion  of  milestones  for  the  software  project 
planning  activities  as  compared  to  the  plan)? 


7.  Does  the  project  manager  review  the  activities  for 
planning  the  software  project  on  both  a  periodic  and 
event-driven  basis? 


Software  Project  Tracking  and  Oversight 


1.  Are  the  project’s  actual  results  (e.g.,  schedule, 
size,  and  cost)  compared  with  estimates  in  the  soft¬ 
ware  plans? 


2.  Is  corrective  action  taken  when  actual  results  devi¬ 
ate  significantly  from  the  project’s  software  plans? 


3.  Are  changes  in  the  software  commitments  agreed 
to  by  all  affected  groups  and  individuals? 


4.  Does  the  project  follow  a  written  organizational 
policy  for  both  tracking  and  controlling  its  software 
development  activities? 


5.  Is  someone  on  the  project  assigned  specific 
responsibilities  for  tracking  software  work  products 
and  activities  (e.g.,  effort,  schedule,  and  budget)? 


6.  Are  measurements  used  to  determine  the  status  of 
the  activities  for  software  tracking  and  oversight 
(e.g.,  total  effort  expended  in  performing  tracking 
and  oversight  activities)? 


7.  Are  the  activities  for  software  project  tracking  and 
oversights  reviewed  with  senior  management  on  a 
periodic  basis  (e.g.,  project  performance,  open 
issues,  risks,  and  action  items)? 


Software  Subcontract  Management 


1.  Is  a  documented  procedure  used  for  selecting  sub¬ 
contractors  based  on  their  ability  to  perform  the 
work? 


2.  Are  changes  to  subcontracts  made  with  the  agree¬ 
ment  of  both  the  prime  contractor  and  the  subcon¬ 
tractor? 
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QUESTIONS 


Project 


3.  Are  periodic  technical  interchanges  held  with  sub¬ 
contractors? 


4.  Are  the  results  and  performance  of  the  software 
subcontractor  tracked  against  the  commitments? 


5.  Does  the  project  follow  a  written  organizational 
policy  for  managing  software  subcontracts? 


6.  Are  the  people  responsible  for  managing  software 
subcontracts  trained  in  managing  software  subcon¬ 
tracts? 


7.  Are  measurements  used  to  determine  the  status  of 
the  activities  for  managing  software  subcontracts 
(e.g.,  schedule  status  with  respect  to  planned  delivery 
dates  and  effort  expended  for  managing  the  subcon¬ 
tract)? 


8.  Are  the  software  subcontract  activities  reviewed 
with  the  project  manager  on  both  periodic  and  event- 
driven  basis? 


Software  Quality  Assurance  (SQA) 


1.  Are  SQA  activities  planned? 


2.  Does  SQA  provide  objective  verification  that  soft¬ 
ware  products  and  activities  adhere  to  applicable 
standards,  procedures,  and  requirements? 


3.  Are  the  results  of  SQA  reviews  and  audits  pro¬ 
vided  to  affected  groups  and  individuals  (e.g.,  those 
who  performed  the  work  and  those  who  are  responsi¬ 
ble  for  the  work)? 


4.  Are  issues  of  noncompliance  that  are  not  resolved 
within  the  software  project  addressed  by  senior  man¬ 
agement  (e.g.,  deviations  from  applicable  stan¬ 
dards)? 


5.  Does  the  project  follow  a  written  organizational 
policy  for  implementing  SQA? 


6.  Are  adequate  resources  provided  for  performing 
SQA  activities  (e.g.,  funding  and  a  designated  man¬ 
ager  who  will  receive  and  act  on  software  noncom¬ 
pliance  items)? 


# 
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Project 


QUESTIONS 


7.  Are  measurements  used  to  determine  the  cost  and 
schedule  status  of  the  activities  performed  for  SQA 
(e.g.,  work  completed,  effort  and  funds  expended 
compared  to  the  plan)? 


8.  Are  activities  for  SQA  reviewed  with  senior  man¬ 
agement  on  a  periodic  basis? 


Software  Configuration  Management  (SCM) 


1.  Are  SCM  activities  planned  for  the  project? 


2.  Has  the  project  identified,  controlled,  and  made 
available  the  software  work  products  through  the  use 
of  software  configuration  management? 


3.  Does  the  project  follow  a  documented  procedure 
to  control  changes  to  configuration  items/units? 


4.  Are  standard  reports  on  software  baselines  (e.g., 
software  configuration  control  board  minutes  and 
change  request  summary  and  status  reports)  distrib¬ 
uted  to  affected  groups  and  individuals? 


5.  Does  the  project  follow  a  written  organizational 
policy  for  implementing  software  configuration  man¬ 
agement  activities? 


6.  Are  project  personnel  trained  to  perform  the  soft¬ 
ware  configuration  management  activities  for  which 
they  are  responsible? 


7.  Are  measurements  used  to  determine  the  status  of 
activities  for  software  configuration  management 
(e.g.,  effort  and  funds  expended  for  software  config¬ 
uration  management  activities)? 


8.  Are  periodic  audits  performed  to  verify  that  soft¬ 
ware  baselines  conform  to  the  documentation  that 
defines  them  (e.g.,  by  the  SCM  group)? 


Organization  Process  Focus 


1.  Are  the  activities  for  developing  and  improving 
the  organization’s  and  project’s  software  processes 
coordinated  across  the  organization  (e.g.,  via  a  soft¬ 
ware  engineering  process  group)? 
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QUESTIONS 


Project 


2.  Is  your  organization’s  software  process  assessed 
periodically? 


3.  Does  your  organization  follow  a  documented  plan 
for  developing  and  improving  its  software  process? 


4.  Does  senior  management  sponsor  the  organiza¬ 
tion’s  activities  for  software  process  development 
and  improvements  (e.g.,  by  establishing  long-term 
plans,  and  by  committing  resources  and  funding)? 


5.  Do  one  or  more  individuals  have  full-time  or  part- 
time  responsibility  for  the  organization’s  software 
process  activities  (e.g.,  a  software  engineering  pro¬ 
cess  group)? 


6.  Are  measurements  used  to  determine  the  status  of 
the  activities  performed  to  develop  and  improve  the 
organization’s  software  process  (e.g.,  effort 
expended  for  software  process  assessment  and 
improvement)? 


7.  Are  the  activities  performed  for  developing  and 
improving  software  processes  reviewed  periodically 
with  senior  management? 


Organization  Process  Definition 


1.  Has  your  organization  developed,  and  does  it 
maintain,  a  standard  software  process? 


2.  Does  the  organization  collect,  review,  and  make 
available  information  related  to  the  use  of  the  organi¬ 
zation’s  standard  software  process  (e.g.,  estimates 
and  actual  data  on  software  size,  effort,  and  cost; 
productivity  data;  and  quality  measurements)? 


3.  Does  the  organization  follow  a  written  policy  for 
both  developing  and  maintaining  its  standard  soft¬ 
ware  process  and  related  process  assets  (e.g., 
descriptions  of  approved  software  life  cycles)? 


4.  Do  individuals  who  develop  and  maintain  the 
organization’s  standards  software  process  receive  the 
required  training  to  perform  these  activities? 
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Project 


QUESTIONS 


5.  Are  measurements  used  to  determine  the  status  of 
the  activities  performed  to  define  and  maintain  the 
organization’s  standard  software  process  (e.g.,  status 
of  schedule  milestones  and  the  cost  of  process  defini¬ 
tion  activities)? 


6.  Are  the  activities  and  work  products  for  develop¬ 
ing  and  maintaining  the  organization’s  standard  soft¬ 
ware  process  subjected  to  SQA  review  and  audit? 


Training  Program 


1.  Are  training  activities  planned? 


2.  Is  training  provided  for  developing  the  skills  and 
knowledge  needed  to  perform  software  managerial 
and  technical  roles? 


3.  Do  members  of  the  software  engineering  group 
and  other  software-related  groups  receive  the  train¬ 
ing  necessary  to  perform  their  roles? 


4.  Does  your  organization  follow  a  written  organiza¬ 
tional  policy  to  meet  its  training  needs? 


5.  Are  adequate  resources  provided  to  implement  the 
organization’s  training  program  (e.g.,  funding,  soft¬ 
ware  tools,  appropriate  training  facilities)? 


6.  Are  measurements  used  to  determine  the  quality 
of  the  training  program? 


7.  Are  training  program  activities  reviewed  with 
senior  management  on  a  periodic  basis? 


Integrated  Software  Management 


1.  Was  the  project’s  defined  software  process  devel¬ 
oped  by  tailoring  the  organization’s  standard  soft¬ 
ware  process? 


2.  Is  the  project  planned  and  managed  in  accordance 
with  the  project’s  defined  software  process? 


3.  Does  the  project  follow  a  written  organizational 
policy  requiring  that  the  software  project  be  planned 
and  managed  using  the  organization’s  standard  soft¬ 
ware  process. 


Project 


QUESTIONS 


4.  Is  training  required  for  individuals  tasked  to  tailor 
the  organization’s  standard  software  process  to 
define  a  software  process  for  a  new  project? 


5.  Are  measurements  used  to  determine  the  effective¬ 
ness  of  the  integrated  software  management  activi¬ 
ties  (e.g.,  frequency,  causes  and  magnitude  of 
replanning  efforts? 


6.  Are  the  activities  and  work  products  used  to  man¬ 
age  the  software  project  subjected  to  SQA  review 
and  audit? 


Software  Product  Engineering 


1.  Are  the  software  work  products  produced  accord¬ 
ing  to  the  project’s  defined  software  process? 


2.  Is  consistency  maintained  across  software  work 
products  (e.g.,  is  the  documentation  tracing  allocated 
requirements  through  software  requirements, 
design,  code,  and  test  cases  maintained)? 


3.  Does  the  project  follow  a  written  organizational 
policy  for  performing  the  software  engineering  activ¬ 
ities  (e.g.,  a  policy  which  requires  the  use  of  appro¬ 
priate  methods  and  tools  for  building  and 
maintaining  software  products)? 


4.  Are  adequate  resources  provided  for  performing 
the  software  engineering  tasks  (e.g.,  funding,  skilled 
individuals,  and  appropriate  tools)? 


5.  Are  measurements  used  to  determine  the  function¬ 
ality  and  quality  of  the  software  products  (e.g.,  num¬ 
bers,  types,  and  severity  of  defects  identified)? 


6.  Are  the  activities  and  work  products  for  engineer¬ 
ing  software  subjected  to  SQA  reviews  and  audits 
(e.g.,  is  required  testing  performed,  are  allocated 
requirements  traced  through  the  software  require¬ 
ments,  design,  code,  and  test  cases)? 


m 

B 

C 

D 

E 

n 

-1 

1 

Project 


QUESTIONS 


Intergroup  Coordination 


1.  On  the  project,  do  the  software  engineering  group 
and  other  engineering  groups  collaborate  with  the 
customer  to  establish  the  system  requirements? 


2.  Do  the  engineering  groups  agree  to  the  commit¬ 
ments  as  represented  in  the  overall  project  plan? 


3.  Do  the  engineering  groups  identify,  track,  and 
resolve  intergroup  issues  (e.g.,  incompatible  sched¬ 
ules,  technical  risks,  or  system-level  problems)? 


4.  Is  there  a  written  organizational  policy  that  guides 
the  establishment  of  interdisciplinary  engineering 
teams? 


5.  Do  the  support  tools  used  by  different  engineering 
groups  enable  effective  communication  and  coordi¬ 
nation  (e.g.,  compatible  word  processing  systems, 
database  systems,  and  problem  tracking  systems)? 


6.  Are  measures  used  to  determine  the  status  of  the 
intergroup  coordination  activities  (e.g.,  effort 
expended  by  the  software  engineering  group  to  sup¬ 
port  other  groups)? 


7.  Are  the  activities  for  intergroup  coordination 
reviewed  with  the  project  manager  on  both  a  periodic 
and  event-driven  basis? 


Peer  Reviews 


1.  Are  peer  reviews  planned? 


2.  Are  actions  associated  with  defects  that  are  identi¬ 
fied  during  peer  reviews  tracked  until  they  are 
resolved? 


3.  Does  the  project  follow  a  written  organizational 
policy  for  performing  peer  reviews? 


4.  Do  participants  of  peer  reviews  receive  the  train¬ 
ing  required  to  perform  their  roles? 
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QUESTIONS 


Project 


5.  Are  measurements  used  to  determine  the  status  of 
peer  review  activities  (e.g.,  number  of  peer  reviews 
performed,  effort  expended  on  peer  reviews,  and 
number  of  work  products  reviewed  compared  to  the 
plan)? 


6.  Are  peer  review  activities  and  work  products  sub¬ 
jected  to  SQA  review  and  audit  (e.g.,  planned 
reviews  are  conducted  and  follow-up  actions  are 
tracked)? 


Quantitative  Process  Management 


1.  Does  the  project  follow  a  documented  plan  for 
conducting  quantitative  process  management? 


2.  Is  the  performance  of  the  project’s  defined  soft¬ 
ware  process  controlled  quantitatively  (e.g.,  through 
the  use  of  quantitative  analytic  methods)? 


3.  Is  the  process  capability  of  the  organization’s  stan¬ 
dards  software  process  known  in  quantitative  terms? 


4.  Does  the  project  follow  a  written  organizational 
policy  for  measuring  and  controlling  the  perfor¬ 
mance  of  the  project’s  defined  software  process  (e.g., 
projects  plan  for  how  to  identify,  analyze,  and  con¬ 
trol  special  causes  of  variations)? 


5.  Are  adequate  resources  provided  for  quantitative 
process  management  activities  (e.g.,  funding,  soft¬ 
ware  support  tools,  and  organizational  measurement 
program)? 


6.  Are  measurements  used  to  determine  the  status  of 
the  quantitative  process  management  activities  (e.g., 
cost  of  quantitative  process  management  activities 
and  accomplishment  of  milestones  for  quantitative 
process  management  activities)? 


7.  Are  the  activities  for  quantitative  process  manage¬ 
ment  reviewed  with  the  project  manager  on  both  a 
periodic  and  event-driven  basis? 


Project 


QUESTIONS 


Software  Quality  Management 


1.  Are  the  activities  for  managing  software  quality 
planned  for  the  project? 


2.  Does  the  project  use  measurable  and  prioritized 
goals  for  managing  the  quality  of  its  software  prod¬ 
ucts  (e.g.,  functionality,  reliability,  maintainability, 
and  usability)? 


3.  Are  measurements  of  quality  compared  to  goals 
for  software  product  quality  to  determine  if  the  qual¬ 
ity  goals  are  satisfied? 


4.  Does  the  project  follow  a  written  organizational 
policy  for  managing  software  quality? 


5.  Do  members  of  the  software  engineering  group 
and  other  software-related  groups  receive  required 
training  in  software  quality  management  (e.g.,  train¬ 
ing  in  collecting  measurement  data  and  benefits  of 
quantitatively  managing  product  quality)? 


6.  Are  measurements  used  to  determine  the  status  of 
the  activities  for  managing  software  quality  (e.g.,  the 
cost  of  poor  quality)? 


7.  Are  the  activities  performed  for  software  quality 
management  reviewed  with  senior  management  on  a 
periodic  basis? 


Defect  Prevention 


1 .  Are  defect  prevention  activities  planned? 


2.  Does  the  project  conduct  causal  analysis  meetings 
to  identify  common  causes  of  defects? 


3.  Once  identified,  are  common  causes  of  defects  pri¬ 
oritized  and  systematically  eliminated? 


4.  Does  the  project  follow  a  written  organizational 
policy  for  defect  prevention  activities? 
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Project 


QUESTIONS 


5.  Do  members  of  the  software  engineering  group 
and  other  software-related  groups  receive  required 
training  to  perform  their  defect  prevention  activities 
(e.g.,  training  in  defect  prevention  methods  and  the 
conduct  of  task  kick-off  or  causal  analysis  meet¬ 
ings)?  _ _ 


6.  Are  measurements  used  to  determine  the  status  of 
defect  prevention  activities  (e.g.,  the  time  and  cost 
for  identifying  and  correcting  defects  and  the  number 
of  action  items  proposed,  open,  and  completed)? 


7.  Are  the  activities  and  work  products  for  defect 
prevention  subjected  to  SQA  review  and  audit? 


Technology  Change  Management 


1 .  Does  the  organization  follow  a  plan  for  managing 
technology  changes? 


2.  Are  new  technologies  evaluated  to  determine  their 
effect  on  quality  and  productivity? 


3.  Does  the  organization  follow  a  documented  proce¬ 
dure  for  incorporating  new  technologies  into  the 
organization’s  standard  software  process? 


4.  Does  senior  management  sponsor  the  organiza¬ 
tion’s  activities  for  managing  technology  change 
(e.g.,  by  establishing  long-term  plans  and  commit¬ 
ments  for  funding,  staffing,  and  other  resources)? 


5.  Do  process  data  exist  to  assist  in  the  selection  of 
new  technology? 


6.  Are  measurements  used  to  determine  the  status  of 
the  organization’s  activities  for  managing  technology 
change  (e.g.,  the  effect  of  implementing  technology 
changes)? 


7.  Are  the  organization’s  activities  for  managing 
technology  change  reviewed  with  senior  manage¬ 
ment  on  a  periodic  basis? 


QUESTIONS 


Process  Change  Management 

1.  Does  the  organization  follow  a  documented  proce¬ 
dure  for  developing  and  maintaining  plans  for  soft¬ 
ware  process  improvement? 

2.  Do  people  throughout  your  organization  partici¬ 
pate  in  software  process  improvement  activities  (e.g., 
on  teams  to  develop  software  process  improve¬ 
ments)? 

3.  Are  improvements  continually  made  to  the  organi¬ 
zation’s  standard  software  process  and  the  projects’ 
defined  software  processes? 

4.  Does  the  organization  follow  a  written  policy  for 
implementing  software  process  improvements? 

5.  Is  training  in  software  process  improvement 
required  for  both  management  and  technical  staff? 

6.  Are  measurements  made  to  determine  the  status  of 
the  activities  for  software  process  improvement  (e.g., 
the  effect  of  implementing  each  process  improve¬ 
ment  compared  to  its  defined  goals)? 

7.  Are  software  process  improvement  efforts 
reviewed  with  senior  management  on  a  periodic 
basis? 


C-16 


APPENDIX  D. 

PRESENTATION  FOR  PRE-PROPOSAL  CONFERENCE 


Software  Capability  Evaluation 


17  October  1994 


SCE  Purpose 


Evaluates  an  offeror’s  software  development  process  at  a  specific 
organization  site 

Measures  the  offeror’s  process  against  defined  industry  accepted 
standards  developed  at  the  Software  Engineering  institute 

-  Capability  Maturity  Modei,  Version  1.1(SEI/CMU  93-TR-24) 

-  Software  Capability  Evaluation,  Version  2.0(SEI/CMU-93*TR-17) 

-  Software  Maturity  Questionnaire,  Version  1.1  (Aprii,  1994) 

Identifies  strengths,  weaknesses,  and  process  improvement  efforts 
associated  with  offeror’s  software  deveiopment  process 
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Capability  Maturity  Model 


Maturity  Level 

Key  Process  Areas 

5 

Optimizing 

Process  Change  Management 

Technology  Change  Management 

Defect  Prevention 

4 

Managed 

Software  Quality  Management 
Quantitative  Process  Management 

3 

Defined 

Peer  Reviews 

Intergroup  Coordination 

Software  Product  Engineering 

Integrated  Software  Management 

Training  Program 

Organization  Process  Definition 
Organization  Process  Focus 

2 

Repeatable 

Software  Configuration  Management 
Software  Quality  Assurance 

Software  Subcontract  Management 
Software  Project  Tracking  And  Oversight 
Software  Project  Planning 

Requirements  Management 

1 

Initial 

“People” 

SCE  Proposal  Requirement 

•  Offerors  asked  to  provide  POC  and  site  location 

Site  of  where  most  of  the  Integration  will  occur 

-  Site  where  most  of  the  critical  software  will  be  developed 

•  Documentation  to  be  submitted  with  proposal 

Project  Profiles  for  4-6  projects 

-  SEI  Maturity  Questionnaire  responses  for  each  project 

-  Software  Process  Improvement  Plan 

•  Criteria  for  projects  selection 

-  Similar  in  scope  to  BM/C3  (e.g.,  domain,  size,  process) 
From  the  same  organization  developing  BM/C3  software 

-  No  “code-word”  classified  projects 

•  SCE  documentation  to  be  appended  to  proposal 
(not  part  of  proposal  page  limit) 
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Initial  Site  Visit  Coordination 


•  SCE  team  pre-coordination  activities  for  site  visit 

-  Provide  at  least  2-day  notice  prior  to  site  visit 

-  Identify  projects,  interview  candidates,  schedule 

-  List  of  general  documentation  to  have  available  at  start  of  SCE 
(e.g.,  Project  A’s  Software  Development  Plan,  SQA  Plan) 

•  Preparation  instructions  for  offeror’s  Overview  presentation 

-  Description  of  organizational  software  development  process 
(e.g.,  standards,  policies,  procedures) 

-  Organization  structure  (project  and  organization  level) 

-  Relationship  of  supporting  departments  (e.g.,  SQA,  training, 
SCM,  process  improvement) 

-  Software  process  improvement  plans  and  recent 
accomplishments 

-  Limited  to  1-2  hours  on  first  day  of  SCE 


General  SCE  Site  Visit  Agenda 


•  Day  1 : 

-  SCE  introductory  briefing  by  government  team 

-  Overview  briefing  by  offeror 

-  Interview:  program  managers 

•  Day  2: 

-  Documentation  review 

-  Exploratory  interviews  (e.g.,  SCM,  SQA,  Standards) 

-  Detailed  documentation  review 

•  Day  3: 

-  Consolidation  interviews 

-  Report  writing 
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Focus  of  Interviews 


Inputs  Objectives 


“Non-attribution” 


SCE  Results 


•  Define  risks  for  key  process  areas  reiative  to  the  defined  levei  (3) 

-  Strengths  of  offeror’s  practices 

-  Weaknesses  of  offeror’s  practices 

-  Process  improvement  activities 

•  Evaiuate  offeror’s  Maturity  Questionnaire  responses 

•  Assess  offeror’s  Software  Process  Improvement  Plan  (if  available) 

•  Results  presented  at  “Disappointed  Offerors  Debrief’’ 


APPENDIX  E. 

SCE  TEAM  ENTRANCE  BRIEFING 


BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Software  Capability  Evaluation  (SCE) 
Team  Entry  Brief 


<Name,  SCE  Team  Lead> 
<Organization  name> 


Conducted  at  the  request  of  the  Ballistic  Missiie  Defense  Office  (BMDO) 


UNCUSSIFIED 


BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Purpose  and  Agenda 


UNCLASSIFIED 
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The  SCE  Team 


BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


SCE  Purpose 


r 

SCE 

- — - \ 

•  Evaluate  Offeror’s  Software  Process  Capability 

Purpose 

_ 

•  Findings  Used  to  Determine  Software  Process 
Related  Risk  to  Acquisition 

Maturity 

Questionnaires 


Proposal 


SCE  Offeror  C 


SCEOHerorB 


SCE 
Offeror  A 


Source 

Selection 


Contract 

Performance 


UNCLASSIFIED 


BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Team  Process  and  Ground  Rules 


Interviews 


>  No  attribution  of  information  obtained  to  an  individuai  or  a 
specific  project 

>  Team  wili  interview  one  individuai  at  a  time 

'  Team  may  interview  an  individuai  more  than  once 
'  interview  scheduie  wiil  become  dynamic  after  first  day 
'  Aii  changes  to  interview  scheduie  made  through  site  visit 
coordinator  _ 


•  The  team  wiii  look  for  objective  evidence  (or  lack  of  objective 
evidence)  to  substantiate  what  it  hears  in  interviews 

•  Aii  documentation  requests  coordinated  through  site 
coordinator 

•  Aii  documentation  returned  at  the  end  of  site  visit;  the  team 
wiii  not  make  any  copies  of  the  documents 


Team  must  observe  supporting  evidence  from  two  or  more 
independent  sources 

'  Hndings  generated  through  consensus  process 


UNCLASSIFIED 
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BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Generic  Site  Visit  Schedule 


8:30  -  9:00 
9:00  - 10:00 
10:00-1:30 
1:30-5:30 
5:30  -  7:30 


7:30  -  9:00 
9:00  - 10:30 
10:30  - 12:30 
12:30-1:30 
1:30-5:00 


DAY  ONE 


Team  Entry  Briefing 

Contractor  Entrance  Briefing 

Document  Review 

Interviews  and  Lunch 

Team  Caucus  and  Document  Review 


Interviews 

Document  Review  and  Lunch 
Interviews 

Team  Caucus  and  Document  Review 


DAY  THREE 


Document  Review 
Team  Meeting 
Consolidation  interviews 
Additional  Document  Reviews/Lunch 
Preparation  of  Findings 


UNCLASSIFIED 


BALLISTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Target  Process  Capability 


Level  5  KPAs 


1 


Technology  Change  Management 


Level  3  KPAs 


Peer  Reviews 
Intergroup  Coordination 
Software  Product  Engineering 
Integrated  Software  Management 
Training  Program 
Organization  Process  Definition 
Organization  Process  Focus 


Level  2  KPAs 


Software  Configuration  Management 
Software  Quality  Assurance 
Software  Subcontract  Management 
Software  Project  Tracking  and  Oversight 
Software  Project  Planning 
Requirements  Management 


Processes  Are  Specified  in  Terms  of  CMM  Key  Process  Areas  (KPAs) 


UNCLASSIFIED 
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Findings  Report 


BALUSTIC 

MISSILE 

DEFENSE 

ORGANIZATION 


Summary  Findings _ 

KPA  acceptability  criteria 
tailored  for  acquisition 

Summary  of  findings  at  KPA 
level 


Detailed  Findings 

Generated  for  each  KPA  in 
target  process  capability 

Strenaths/weaknesses/ 
notedimprovement  activities 


Findings  Report 

Findings  Summary  I  KPA  Name 


Acceptable: 

KPAs 


KPA  Summary 
Statement 

Strengths: 


Could  benefit  from  Weaknesses: 
improvement: 


KPAs 


Noted  Improvement 
Activities: 


UNCLASSIFIED 


APPENDIX  R 

NOTICE  TO  CONTRACTOR 


\ViA 

COMPUTER  AND  SOFTWARE 
ENGINEERING  DIVISION 

Date 


MEMORANDUM  FOR  BMC3  Offeror’s  SCE  Point  of  Contact 

FROM:  Beth  Springsteen 

SUBJECT:  Software  Capability  Evaluation  Site  Visit 


The  BMC3  Software  Capability  Evaluation  (SCE)  will  be  conducted  at _ 

over  the  course  of  three  days  next  week,  Tuesday  through  Thursday, _ .  Following  is 

a  list  of  information  that  should  be  prepared  for  the  SCE  site  visit. 

Secnritv  Clearance:  If  the  evaluation  team  needs  to  have  a  security  clearance  on  file  at  your 
facility,  please  send  my  secretary  the  appropriate  information  (i.e.,  fax  number  of  security  office, 
phone  number  of  security  office).  This  is  for  convenience  on  the  part  of  the  evaluation  team  so  that 
they  may  go  to  the  rest  rooms  and  enter  and  exit  the  building  as  needed.  My  secretary’s  name  is 
Libby  Gonzalez:  Fax  number  is  (703)  845-6848,  phone  number  is  (703)  845-6612. 

Interview  Room:  The  SCE  team  will  need  a  closed  meeting  room  capable  of  handling  at 
least  eight  people  during  the  site  visit.  The  room  should  also  contain  two  tablets  on  easels  with 
markers.  The  team  will  also  need  access  to  a  copier  machine  and  phone. 

Site  Visit  Schedule:  Attached  is  a  generic  schedule  for  the  three-day  site  visit.  Identified  are 
typical  titles  of  the  individuals  who  will  be  scheduled  for  interviews  at  this  time.  The  interviewees 
are  in  a  standard  order  but  this  can  be  altered  in  order  to  accommodate  their  schedules.  However, 
keep  the  allocated  length  of  the  interview  as  identified. 

Entrance  Briefing:  At  8:30  on  the  first  day,  the  BMC3  SCE  team  will  give  a  brief  introduc¬ 
tion  to  the  site  visit  activities.  In  addition,  each  offeror  is  requested  to  provide  a  brief  introduction 
of  his  or  her  software  development  process  and  organization.  Provide  five  copies  of  the  presenta¬ 
tion  material  to  the  SCE  team.  This  presentation  should  not  exceed  one  hour  and  should  include  the 
following: 

•  Overview  of  organization  structure  (selected  project  and  organizational  level  groups 
that  support  the  projects) 

•  Responsibilities  of  organizational-level  groups  which  support  the  software-intensive 
projects  (e.g.,  groups  responsible  for  software  process  improvement,  software  quality 
assurance,  software  configuration  management,  training  program,  development  of  soft¬ 
ware  policies  and  standards,  software  technology  change  management) 

•  Overview  of  organization’s  software  development  process  (e.g.,  standards,  policies,  and 
procedures) 


nnmmp.nt  Request:  Attached  is  a  list  of  documents  to  have  available  when  the  SCE  team 
arrives  on  site  (these  may  be  placed  in  the  interview  room).  In  addition,  documents  that  were  ref¬ 
erenced  in  response  to  the  Maturity  Questionnaire  should  be  made  available  at  the  beginning  of  the 
site  visit.  These  references  should  be  organized  by  each  selected  project  and  by  question  in  the 
Maturity  Questionnaire.  Other  documents  will  be  requested  during  the  course  of  interviews  as  nec¬ 
essary. 

Interview  Candidates:  Attached  is  a  list  of  the  interview  candidates  selected  from  the  orga¬ 
nization  charts  which  were  submitted  with  the  proposals.  Since  the  offeror  s  job  titles  may  be 
unique  to  that  organization  and  misinterpreted  by  the  SCE  team,  please  provide  input  in  the  event 
there  may  be  a  mismatch.  In  the  left  column  is  the  generic  title  that  the  SCE  team  is  using  to  char¬ 
acterize  typical  project  responsibilities.  To  the  right  is  the  selected  interview  candidate  with  their 
job  title  as  it  appears  in  the  organization  chart.  The  SCE  team  has  made  some  assumptions  that 
these  responsibilities  match  (e.g.,  software  quality  assurance  vs.  product  assurance).  Coordinate 
with  the  SCE  team  at  the  beginning  of  the  site  visit  to  ensure  it  interviews  the  most  appropriate 
candidate. 

In  addition,  several  job  titles  were  not  easily  identified  in  the  offeror’s  organization  charts 
that  the  SCE  team  would  like  to  interview.  These  are  typically  the  departments  that  may  exist  at 
the  organizational  level  rather  than  the  project  level.  If  such  individuals  or  groups  exist,  please 
identify  and  schedule  those  responsible  for  managing  a)  software  process  improvement  across 
projects,  b)  software  configuration  management  across  projects,  and  c)  software  quality  assurance 
across  projects. 

The  interview  schedule  and  the  interview  candidates  may  change  during  the  course  of  the 
site  visit. 


Attachments: 

•  Generic  Site  Visit  Schedule 

•  General  Document  List 

•  Interview  Candidates 


Generic  Site  Visit  Schedule 


Day  One: 

•  7:30-8:30  SCE  team  arrives  on  site 

•  8:30-9:00  SCE  team  introduction  briefing  to  contractor 

•  9:00-10:00  Contractor  entrance  briefing 

•  10:00- 1 2:30  Documentation  review 

•  12:30-1:30  Lunch 

•  1:30-5:30  Interviews 

Hrs  1.25  Software  manager  (project  1) 

0.25  Break 

0.50  Project  manager  (project  1) 

0.25  Break 

1.00  Manager  of  software  process  improvement 

0.25  Break 

0.50  SQA  manager 

•  5:30-7:30  End-of-day  caucus 

•  7:30+  Document  review  at  hotel 


Day  Two: 


7:30-8:30  Document  review 
8:30-12:00  Interviews 
Hrs  0.50  SQA  (project  2) 

0.25  Break 

0.50  Subcontractor  software  manager  (project  2) 
0.25  Break 

0.50  System  engineer  (Project  1) 

0.25  Break 
0.50  SCM  manager 
0.25  Break 

0.50  Test  Engineer  (project  2) 

12:00-2:00  Document  review  with  lunch 
2:00-5:00  Interviews 

Hrs  0.75  Developer/Group  lead  (project  2) 

0.25  Break 

1.25  Software  manager  (project2) 

0.25  Break 

0.50  SCM  (project  1) 


Generic  Site  Visit  Schedule  (cont.) 


Day  Three: 

•  7:30-9:00  Document  review 

•  9:00-10:30  Team  meeting/consolidation  plan 

•  10:30-12:30  Interviews 

Hrs  0.50  Developer/Group  lead  (project  1) 

0.50  Software  manager  (project  3) 

0.25  Break 
0.75  other 

•  12:30- 1 :30  Additional  documentation  review  with  lunch 

•  1 :30-5:00  Preparation  of  findings 


General  Document  List 

(Provided  documents  exist  and  are  being  used) 


1 .  Division-level  documents  (generally  apply  across  all  projects) 

•  Software  process  improvement  plan 

•  Software  policy,  standards,  and  procedures  (2  copies) 

•  Generic  software  development  plan 

•  Software  quality  assurance  procedures/plan 

•  Software  configuration  management  procedures/plan 

2.  Project  documents  (for  all  projects  selected  for  review  by  SCE  team) 

•  Program  management  plan 

•  Software  development  plan 

•  SCM  plan 

•  SQA  plan 

•  Software  test  procedures 

•  Software  standards  and  procedures  manual 

•  Sample  software  development  folder 


Interview  Candidates 


Project  1:  (name) 

•  Project  manager:  _ _ 

•  Software  manager:  _ _ 

•  Project  software  configuration  manager: 

•  System  engineer: _ 

•  Developer/Group  lead: _ 


Project  2:  (name) 

•  Software  manager: _ 

•  Manager  of  software  subcontractor: 

•  Project  software  quality  assurance: 

•  Software  test  engineer: _ 

•  Developer/Group  lead: _ 


Project  3:  (name) 

•  Software  manager: _ _ 

Organization-wide  department  heads: 

•  Manager  of  software  process  improvement: _ 

•  Manager  of  software  quality  assurance: _ 

•  Manager  of  software  configuration  management: 


APPENDIX  G. 

INTERVIEW  INTRODUCTION 


DRAFT 


8/3/95 


The  following  points  should  be  covered  by  the  team  leads  at  the  initiation  of  each 
interview  [Ragan  1994]: 

•  Introduce  yourself  and  set  a  friendly  tone  (there  is  typically  not  time  to  intro¬ 
duce  the  entire  team).  Explain  that  the  team  is  trying  to  gain  an  understanding 
of  the  organization’s  software  process  and  the  interviewee’s  roles  and  responsi¬ 
bilities  in  the  organization. 

•  Put  the  interviewee  at  ease  by  emphasizing  that  the  interview  process  is  non¬ 
attribution  and  that  the  sources  of  data  gathered  during  the  interviews  will 
remain  anonymous. 

•  Explain  that  several  team  members  will  be  asking  questions,  and  that  interrup¬ 
tions  may  occur.  Ask  that  the  interviewee  not  be  offended  by  interruptions  since 
interruptions  may  be  required  to  meet  the  interview  objectives  in  the  allotted 
time. 

•  If  the  interviewee  does  not  know  the  answer  to  a  question,  ask  him  or  her  to 
identify  the  appropriate  person  to  ask  (or  document  to  review). 

•  Let  the  interviewee  know  the  team  will  probably  be  requesting  to  see  documen¬ 
tation  that  will  help  the  team’s  understanding  of  the  processes  described.  Ask 
the  interviewee  to  deliver  all  requested  documentation  to  the  organization’s 
SCE  point  of  contact,  and  let  him  or  her  know  the  documentation  will  be 
returned  once  the  team  has  completed  its  evaluation. 

•  Tell  the  interviewee  that  you  appreciate  his  or  her  time,  assistance,  and  cooper¬ 
ation. 

•  Be  sure  to  thank  the  interviewee  at  the  end  of  the  interview  and  review  the  list 
of  documents  that  were  requested  and  recorded  by  the  SCE  document  tracker. 


APPENDIX  H. 

DOCUMENT  TRACKING  FORM 


APPENDIX  1. 
SAMPLE  FINDINGS 


This  appendix  contains  sample  findings  for  levels  2-3  KPAs  and  Technology 

Change  Management. 

1.  Software  Project  Planning 

1.1  Sample  Strengths 

a.  Software  development  plans  are  developed  according  to  a  documented  proce¬ 
dure  (e.g.,  project  standards  and  procedures,  division-level  Standard  Software 
Development  Process,  Model  Software  Development  Plan). 

b.  Software  size,  effort,  schedule,  cost,  and  computer  resources  estimates  are 
developed  in  accordance  with  documented  procedures  (e.g..  Procedure  for  Esti¬ 
mating  Software  Sizing  and  Timing,  Procedure  for  Estimating  Software  Cost 
and  Schedule,  Software  Cost  Estimating  Guidebook). 

c  Software  group  participates  in  planning  activities  associated  with  the  proposal 
team  (e.g.,  SOW,  WBS,  SDP). 

d.  Long-term  historical  data  are  used  in  developing  software  estimates. 

1.2  Sample  Weaknesses 

a.  Little  evidence  is  available  that  project-level  software  development  plans  are 
reviewed,  managed,  or  controlled. 


2.  Requirements  Management 

2.1  Sample  Strengths 

a.  Requirements  and  changes  to  requirements  are  documented,  reviewed,  and  con¬ 
sistently  controlled  (e.g.,  ECPs  incorporated  via  CCB-controlled  STRs). 

b.  Traceability  tools  are  used  for  managing  allocated  requirements  and  related 
tests  (e.g.,  TRACER  requirements  traceability  tool). 

c.  Software  engineering  group  is  actively  involved  in  allocation  of  system  require¬ 
ments  to  software. 

2.2  Sample  Weaknesses 

a.  Projects  lack  guidance  for  selecting  COTS  software  to  satisfactory  require¬ 
ments. 

b.  Instances  occur  of  inefficient  and  potentially  error-prone  requirements  trace- 
ability  methods  (i.e.,  both  automated  and  manual  wiAin  same  project). 

c.  Projects  lack  guidance  for  selecting  COTS  software  to  satisfy  requirements. 


3.  Software  Project  Tracking  and  Oversight 

3.1  Sample  Strengths 

a.  Cost/schedule  status  is  tracked  through  periodic  reports  (e.g.,  earned  value). 

b.  Risks,  sizes,  efforts,  costs,  schedules  are  tracked  against  plans  (e.g.,  gant  charts, 
fowler  charts,  activity  networks,  C/SCSC,  rate  charts). 

c.  Project  status  is  reviewed  weekly  with  task  managers  and  monthly  with  senior 
management  (e.g.,  weekly  status  meetings,  tracking  book  reviews,  highlights). 

d.  Problems  are  documented  (e.g.,  open/closed  STRs)  and  tracked  to  closure. 

e.  Corrective  actions  are  taken  and  managed  to  closure  when  actual  results  and 
performance  deviate  significantly  from  the  software  plans. 

3.2  Sample  Weaknesses 

a.  No  evidence  is  found  of  software  development  plan  revisions. 

b.  Limited  use  is  made  of  metrics  to  identify  need  for  corrective  actions  and  revi¬ 
sions  to  plans. 

c.  Process  for  action  item  identification  and  tracking  to  closure  is  extremely  infor¬ 
mal  (e.g.,  issues  identified  through  internal  status  meetings  with  PM,  APM, 
SPMs — no  minutes,  no  action  item  lists). 

d.  Instances  occur  of  schedule  milestones  being  defined  by  management  without 
input  from  those  responsible  for  executing  the  task. 

4.  Software  Subcontract  Management 

4.1  Sample  Strengths 

a.  Subcontractor  technical  and  management  personnel  actively  participate  in 
project  meetings  that  address  status,  risk,  and  technical  issues. 

b.  Subcontractor  software  engineering  capability  is  evaluated  prior  to  subcontrac¬ 
tor  selection. 

c.  Subcontractor  software  effort  is  monitored  against  software  development  plan 
prepared  by  subcontractor. 

d.  Documented  procedures  exist  for  planning  subcontract  work  and  for  selecting 
and  monitoring  subcontractor. 

e.  SOW  detailing  commitments  regarding  deliverables,  standards,  statusing  mech¬ 
anisms,  and  other  relevant  items  are  agreed  to  by  both  parties. 

4.2  Sample  Weaknesses 

a.  Verification  of  subcontractor  compliance  with  project-specific  selection  criteria 
is  not  evident 

b.  Procedures  are  insufficient  used  to  verify  organization’s  capability  to  select 
qualified  subcontractors  and  monitor  subcontractor  efforts  effectively  (when 
subcontracted  effort  is  not  co-located). 
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c.  Organization  has  never  selected  or  managed  a  software  development  subcon¬ 
tractor;  therefore,  no  evaluation  of  this  KPA  was  possible. 

5.  Software  Quality  Assurance 

5.1  Sample  Strengths 

a.  Audits  of  software  products  and  processes  are  scheduled  and  conducted 
throughout  the  life-cycle  (e.g.,  walkthroughs,  audits). 

b.  Results  of  product  and  process  evaluations  are  communicated  to  responsible 
engineers,  project  managers,  and  quality  manager. 

c.  Organizational-level  SQA  procedures,  practices,  and  standards  exist  and  are 
augmented  as  needed  on  individual  projects. 

5.2  Sample  Weaknesses 

a.  No  mechanism  exists  to  determine  if  audits  are  representative  and  sufficient. 

b.  SQA  personnel  are  also  performing  some  line  activities  (e.g.,  SCM,  testing). 

c.  No  feedback  mechanism  exists  for  improving  software  quality  process. 

6.  Software  Configuration  Management 

6.1  Sample  Strengths 

a.  Changes  to  baselines  and  the  release  of  software  products  built  from  the  soft¬ 
ware  baseline  library  are  systematically  controlled. 

b.  Configuration  control  of  requirements,  design,  software,  and  associated  docu¬ 
mentation  is  well  defined. 

c.  Tools  support  multiple  levels  of  software  configuration  management  control 
tied  to  development  stages  for  software  libraries. 

d.  CCBs  are  established  to  manage  formal  and  informal  baselines  (e.g..  Software 
CCB  and  Program  CCB). 

e.  Formal  mechanism  is  in  place  for  tracking  problem  reports. 

f.  Regression  testing  is  performed  as  applicable  after  incorporation  of  corrections 
to  code. 

6.2  Sample  Weaknesses 

a.  Insufficient  documentation  exists  of  software  development  library,  roles,  and 
responsibilities 

b.  Regression  testing  is  not  performed  and  reported. 

c.  Traceability  between  requirements,  design,  and  code  is  inadequate. 

d.  No  organization-wide  policies  and  procedures  exist  for  developmental  SCM 
practices. 

e.  No  feedback  mechanism  exists  to  track  and  refine  SCM  practices. 
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7.  Organization  Process  Focus 

7.1  Sample  Strengths 

a  Mechanisms  exist  to  inform  software  groups  at  project  level  of  organization- 
wide  process  improvements  (e.g.,  SSC  Newsletter,  change  pages,  formal  train¬ 
ing). 

b.  A  CCB  and  STRs  are  used  to  control  updates  to  the  organization  software  pro¬ 
cess. 

Organization  metrics  are  used  to  perform  process  improvement  trend  analysis. 

d  Self-assessments  of  the  organization-wide  software  process  are  conducted  to 
identify  strengths  and  weaknesses  (e.g.,  1998, 1991,  planned  in  Apnl  1995). 

e.  Software  process  development  and  improvement  activities  are  coordinated  and 
disseminated  across  the  organization  and  the  projects  (i.e.,  SEPG  has  broad 
membership,  select  business  units  have  SEPGs,  VP  GM  receives  monthly 
SEPG  status). 

7.2  Sample  Weaknesses 

a.  Minimal  feedback  from  projects  occurs  to  identify  opportunities  for  process 
improvements  (e.g..  Phase  II  training  interviews). 

b.  Information  related  to  the  use  of  the  organization  standard  process  by  the 
projects  is  not  collected  and  reviewed  by  SEPG  (e.g.,  project  SDPs  not  consis¬ 
tently  reviewed  by  SEPG). 

c.  Lack  of  organization  metrics  used  to  analyze  process  improvement  needs  and  to 
track  effectiveness  of  improvements  (e.g.,  PRA  reports  collected  but  not  ana¬ 
lyzed). 

8.  Organization  Process  Definition 

8.1  Sample  Strengths 

a.  A  library  of  software  process-related  documentation  is  established,  maintained, 
and  populated  by  the  projects  (e.g.,  SDPs,  metrics  reports,  managernent  plans, 
requirements,  designs,  and  test  specifications/materials,  source  listings,  stan¬ 
dards). 

b.  The  organization’s  standard  software  process  covers  the  entire  software  life- 
cycle  and  includes  the  approved  engineering  methods  and  techniques,  products, 
and  practices/procedures/entry/exit  criteria  for  each  life-cycle  phase. 

c.  Additions  and  revisions  to  the  organization’s  software  process  are  reviewed  and 
approved  by  senior  management. 

8.2  Sample  Weaknesses 

a.  Guidance  for  tailoring  the  organization’s  standard  process  is  not  documented 
for  project  use. 

b.  A  library  of  software  process-related  documentation  is  established,  but  it  lacks 
lessons-leamed  data  and  is  only  populated  by  a  small  set  of  projects. 


c.  Information  related  to  use  of  the  organization ’s  standard  software  process  by  the 
projects  is  not  routinely  collected  and  reviewed  (i.e.,  process  audits  immature). 

d.  A  formal  centralized  software  process  database  and  procedures  to  collect  and 
make  available  data  on  software  processes  and  products  do  not  exist. 

9.  Training  Program 

9.1  Sample  Strengths 

a.  Training  program  on  organizational  processes  is  widely  received  (e.g.,  engi¬ 
neering  process  improvement  training  Phase  1  and  2). 

b.  Organizational  training  plan  exists  and  specifies  required  and  recommended 
training  for  all  grade  levels  within  each  software  discipline. 

c.  Training  records  are  located  in  a  central  repository  and  indicate  progress 
towards  attending  required  classes. 

d.  Information  regarding  organization  training  events  is  disseminated  (e.g., 
monthly  training  calendars,  electronic  mail,  bulletins  on  a  quarterly  and  “as 
available”  basis). 

9.2  Sample  Weaknesses 

a.  Existence  of  training  plans  is  not  apparent  for  some  projects,  and  those  project 
plans  observed  are  deemed  inadequate  due  to  lack  of  maintenance. 

b.  Required  training  is  typically  provided  only  during  off  hours. 

c.  Lack  of  a  centralized  repository  for  training  records  hampers  planning  and 
tracking. 

d.  Training  for  professional  development  is  left  to  employee’s  initiatives  with 
some  informal  input  from  supervisor. 

10.  Integrated  Software  Management 

10.1  Sample  Strengths 

a.  Software  risks  are  identified,  assessed,  documented,  and  managed  across 
projects  according  to  documented  procedures. 

b.  The  project’s  software  development  process  is  developed  by  tailoring  the  orga¬ 
nization’s  standard  software  development  process  to  reflect  project  needs. 

c.  Organizational  database  is  used  to  support  software  project  planning  and  esti¬ 
mating  (e.g.,  project  control  database). 

10.2  Sample  Weaknesses 

a.  The  ciurent  projects’  defined  software  processes  are  not  tailored  versions  of  the 
current  organization’s  standard  software  process. 

b.  No  evidence  exists  of  formalized  proactive,  risk  management  processes  or  pro¬ 
cedures. 


c.  Organizational  database  to  support  software  project  planning  and  estimating  is 
immature  (e.g.,  metrics  database,  data  center  is  not  automated  and  is  not  always 
used;  it  is  repository  for  project-level  metrics  reports). 

d.  Thresholds  are  not  established  for  parameters;  when  exceeded,  action  is 
required. 

11.  Software  Product  Engineering 

11.1  Sample  Strengths 

a.  Acceptability  and  quality  of  the  software  are  enhanced  by  integrated  software 
engineering  activities  (e.g.,  requirements,  analysis,  design,  code,  and  test). 

b  Common  methods  and  tools  are  defined  and  used  to  support  software  engineer¬ 
ing  activities  (e.g.,  OOA/OOD,  TEAMWORK,  RVTM  traceability  tool.  Struc¬ 
tured  Analysis  and  Design). 

c.  Appropriate  methods  and  tools  are  used  for  conducting  requirements  analysis 
and  design  (e.g.,  Hatley-Pirbhai  method  for  requirements  analysis,  software 
through  pictures  CASE  tool,  structured  analysis  and  design). 

11.2  Sample  Weaknesses 

a.  No  formal  process  exists  for  identifying,  selecting,  and  evaluating  COTS  tools 
for  project  use  along  with  reuseable  software  components. 

b.  Automated  tools  are  not  used  for  conducting  requirements  analysis  and  design 
(e.g.,  requirements  traceability,  design  specification  tools). 

c.  Software  engineering  tool  selection  is  customer  driven  vs.  what  is  appropriate 
to  the  task  and  available  resources  (e.g.,  skill  base,  ease-of-use,  availability). 

12.  Intergroup  Coordination 

12.1  Sample  Strengths 

a.  Multi-discipline  risk  management  team  provides  mechanism  for  tracking  and 
resolving  risks. 

b.  Standardized  process  for  review  of  work  products  by  all  affected  engineering 
groups  ensures  that  products  meet  receiving  group’s  needs. 

c.  New  policy  and  procedures  for  implementation  of  integrated  product  develop¬ 
ment  teams  encourage  integration  of  customer  into  all  phases  of  software  devel¬ 
opment  process  (new  proc^ures  not  yet  institutionalized). 

d.  Working  groups  established  to  address  technical  intergroup  issues  and  resulting 
action  items  are  tracked  to  closure. 

e.  A  master  schedule  identifies  intergroup  commitments  and  is  used  to  coordinate 
and  track  the  work  performed. 

12.2  Sample  Weaknesses 

a.  Customer  interface  is  limited  and  primarily  occurs  during  reviews  of  contractu¬ 
ally  deliverable  documents  and  attendance  at  formal  reviews  (e.g.,  PDR,  CDR). 


b.  No  mechanism  exists  for  regular  communication  between  disciplines  at  work¬ 
ing  level;  coordination  of  activities  performed  by  different  engineering  groups 
within  project  is  accomplished  at  periodic  (e.g.,  weekly)  management  meetings. 

c.  No  evidence  exists  consistent  tracking  of  dependencies  between  tasks  that  are 
performed  by  different  engineering  groups. 

13.  Peer  Reviews 

13.1  Sample  Strengths 

a.  Inspection  action  items  are  documented,  tracked  to  closure,  and  signed  off. 

b.  Statistics  on  the  inspection  process  are  collected,  reported  to  the  SEPG,  and 
used  to  measure  inspection  ROI  and  to  make  inspections  more  effective. 

c.  Peer  reviews  are  scheduled  and  conducted  for  100%  of  software  design,  code, 
and  test  products  for  development  projects. 

d.  Policy  and  procedures  exist  to  support  the  walkthrough  process  at  all  phases  of 
the  software  life  cycle  (requirements,  design,  code). 

13.2  Sample  Weaknesses 

a.  Most  code  peer  reviews  are  desk  reviews  performed  by  one  peer.  This  decreases 
the  opportunities  for  synergistic  problem  detection  and  for  developing  wide¬ 
spread  understanding  of  software  work  products,  and  increases  the  potential  for 
rote,  rabber-stamp  peer  review  efforts. 

b.  Organizational  standard  procedure  for  performing  peer  reviews  is  not  docu¬ 
mented. 

c.  Rigor  of  tracking  action  items  to  closure  varies  across  projects. 

d.  Structured  peer  review  process  not  uniformly  applied  to  software  products  other 
than  PDL  and  code. 

14.  Technology  Change  Management 

14.1  Sample  Strengths 

a.  Selected  software  techniques  are  being  evaluated  (e.g.  software  environment, 
object-oriented  technology,  CASE  tools,  reuse,  domain  analysis). 

b.  Responsibilities  and  resources  are  assigned  for  evaluating  new  software  tools 
and  technologies  (e.g.,  working  and  departments). 

c.  Technology  planning  committee  and  core  technology  teams  are  established  at 
division  level  to  manage  engineering  technology  change  activities,  including 
software. 

d.  Regular  technology  transfer  forums  are  conducted  by  SEPG  to  provide  informa¬ 
tion  on  state-of-the-art  software  technologies/methodologies  (e.g.,  several  ses¬ 
sions  held  on  COTS). 

e.  Organization ’s  standard  process  defines  detailed  process,  criteria,  and  checklists 
to  be  used  for  evaluation  and  selection  of  COTS. 
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14.2  Sample  Weaknesses 

a.  No  formal  organizational  plan  exists  which  captures  process  for  evaluation  and 
infusion  of  new  technologies  into  the  organization’s  standard  process. 

b.  Software  technology  database  containing  status  of  new  tools  and  technologies 
has  been  recently  established  but  is  not  yet  populated  or  used  by  projects. 

c.  No  evidence  exists  of  expertise  available  to  support  project’s  implementation  of 
advance  software  technologies  and  tools. 
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